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Abstract 

We study an online decision making problem where on each round a learner chooses a list of items based 
on some side information, receives a scalar feedback value for each individual item, and a reward that is 
linearly related to this feedback. These problems, known as contextual semibandits, arise in crowdsourcing, 
recommendation, and many other domains. This paper reduces contextual semibandits to supervised 
learning, allowing us to leverage powerful supervised learning methods in this partial-feedback setting. Our 
first reduction applies when the mapping from feedback to reward is known and leads to a computationally 
efficient algorithm with near-optimal regret. We show that this algorithm outperforms state-of-the-art 
approaches on real-world leaming-to-rank datasets, demonstrating the advantage of oracle-based algorithms. 
Our second reduction applies to the previously unstudied setting when the linear mapping from feedback to 
reward is unknown. Our regret guarantees are superior to prior techniques that ignore the feedback. 


1 Introduction 

Decision making with partial feedback, motivated by applications including personalized medicine [22] and 
content recommendation [17], is receiving increasing attention from the machine learning community. These 
problems are formally modeled as learning from bandit feedback, where a learner repeatedly takes an action 
and observes a reward for the action, with the goal of maximizing reward. While bandit learning captures 
many problems of interest, several applications have additional structure: the action is combinatorial in nature 
and more detailed feedback is provided. For example, in internet applications, we often recommend sets 
of items and record information about the user’s interaction with each individual item (e.g., click). This 
additional feedback is unhelpful unless it relates to the overall reward (e.g., number of clicks), and, as in 
previous work, we assume a linear relationship. This interaction is known as the semibandit feedback model. 

Typical bandit and semibandit algorithms achieve reward that is competitive with the single best fixed 
action, i.e., the best medical treatment or the most popular news article for everyone. This is often inadequate 
for recommendation applications: while the most popular articles may get some clicks, personalizing content 
to the users is much more effective. A better strategy is therefore to leverage contextual information to learn 
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Algorithm 

Regret 

Oracle Calls 

Weights w* 

VCEE (Thm. 1) 

y/KLT log N 

r3/2^A:/(LiogiV) 

known 

e-Greedy (Thm. 3) 

{LTf/^(K\ogNf/^ 

1 

known 

Kale etal. [13] 

y/KLT log N 

not oracle-based 

known 

EELS (Thm. 2) 

{LTflog NY 

1 

unknown 

Agarwal et al. [ 1 ] 

Ly^K^TlogN 

^K^T/log N 

unknown 

Swaminathan et al. [24] 

L4/3T2/3(A:iogiV)l/3 

1 

unknown 


Table 1: Comparison of contextual semibandit algorithms for arbitrary policy classes, assuming all rankings 
are valid composite actions. The reward is semibandit feedback weighted according to w*. For known 
weights, we consider w* = 1; for unknown weights, we assume ||ui *||2 < 0{-\fL). 


a rich policy for selecting actions, and we model this as contextual semibandits. In this setting, the learner 
repeatedly observes a context (user features), chooses a composite action (list of articles), which is an ordered 
tuple of simple actions, and receives reward for the composite action (number of clicks), but also feedback 
about each simple action (click). The goal of the learner is to find a policy for mapping contexts to composite 
actions that achieves high reward. 

We typically consider policies in a large but constrained class, for example, linear learners or tree 
ensembles. Such a class enables us to learn an expressive policy, but introduces a computational challenge 
of finding a good policy without direct enumeration. We build on the supervised learning literature, which 
has developed fast algorithms for such policy classes, including logistic regression and SVMs for linear 
classifiers and boosting for tree ensembles. We access the policy class exclusively through a supervised 
learning algorithm, viewed as an oracle. 

In this paper, we develop and evaluate oracle-based algorithms for the contextual semibandits problem. 
We make the following contributions: 

1. In the more common setting where the linear function relating the semibandit feedback to the reward 

is known, we develop a new algorithm, called VCEE, that extends the oracle-based contextual bandit 
algorithm of Agarwal et al. [1]. We show that VCEE enjoys a regret bound between 0{^KLT log 7V) 
and 0{L\/KT log iV), depending on the combinatorial structure of the problem, when there are T rounds 
of interaction, K simple actions, N policies, and composite actions have length L.' VCEE can handle 
structured action spaces and makes calls to the supervised learning oracle. 

2. We empirically evaluate this algorithm on two large-scale learning-to-rank datasets and compare with 
other contextual semibandit approaches. These experiments comprehensively demonstrate that effective 
exploration over a rich policy class can lead to significantly better performance than existing approaches. 
To our knowledge, this is the first thorough experimental evaluation of not only oracle-based semibandit 
methods, but of oracle-based contextual bandits as well. 

3. When the linear function relating the feedback to the reward is unknown, we develop a new algorithm 

called EELS. Our algorithm first learns the linear function by uniform exploration and then, adaptively, 
switches to act according to an empirically optimal policy. We prove an O log regret 

bound by analyzing when to switch. We are not aware of other computationally efficient procedures with 
a matching or better regret bound for this setting. 

See Table 1 for a comparison of our results with existing applicable bounds. 

'Throughout the paper, the O(-) notation suppre.ssed factors polylogarithmic in K, L, T and log N. We analyze finite policy classes, 
hut our work extends to infinite classes by standard discretization arguments. 
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Related work. There is a growing body of work on combinatorial bandit optimization [5, 2] with consid¬ 
erable attention on semibandit feedback [11, 13, 7, 20, 14]. The majority of this research focuses on the 
non-contextual setting with a known relationship between semibandit feedback and reward, and a typical 
algorithm here achieves an 0{y/KLT) regret against the best fixed composite action. To our knowledge, 
only the work of Kale et al. [ 1 3] and Qin et al. [20] considers the contextual setting, again with known 
relationship. The former generalizes the Exp4 algorithm [3] to semibandits, and achieves 0{\/KLT) regret,^ 
but requires explicit enumeration of the policies. The latter generalizes the LinUCB algorithm of Chu et al. 
[8] to semibandits, assuming that the simple action feedback is linearly related to the context. This differs 
from our setting: we make no assumptions about the simple action feedback. In our experiments, we compare 
VCEE against this LinUCB-style algorithm and demonstrate substantial improvements. 

We are not aware of attempts to learn a relationship between the overall reward and the feedback on 
simple actions as we do with EELS. While EELS uses least squares, as in LinUCB-style approaches, it does 
so without assumptions on the semibandit feedback. Crucially, the covariates for its least squares problem are 
observed after predicting a composite action and not before, unlike in LinUCB. 

Supervised learning oracles have been used as a computational primitive in many settings including active 
learning [12], contextual bandits [21, 25, 1, 10], and structured prediction [9]. 

2 Preliminaries 

Let X be a space of contexts and A a set of K simple actions. Let 11 C (X —>■ be a finite set of policies, 
|n| = W, mapping contexts to composite actions. Composite actions, also called rankings, are tuples of L 
distinct simple actions. In general, there are K\/{K — L) \ possible rankings, but they might not be valid 
in all contexts. The set of valid rankings for a context x is defined implicitly through the policy class as 
{7r(a;)}^gn- 

Let A(n) be the set of distributions over policies, and A< (11) be the set of non-negative weight vectors 
over policies, summing to at most 1, which we call subdistributions. Let !(•) be the 0/1 indicator equal to 1 if 
its argument is true and 0 otherwise. 

In stochastic contextual semibandits, there is an unknown distribution T) over triples (a;, y, ^), where x 
is a context, y G [0,1]^ is the vector of reward features, with entries indexed by simple actions as y{a), 
and ^ G [—1,1] is the reward noise, E[^|x, y] = 0. Given y G and A — (oi,..., ol) G A^, we write 
y{A) G for the vector with entries y{ai). The learner plays a T-round game. In each round, nature draws 
{xt, yt,^t) ^ 2) and reveals the context xt- The learner selects a valid ranking At = (at.i) ai, 2 j • • • j 0‘t,L) 
and gets reward rt{At) = i>t£yt{at,i) + Ct. where w* G is a possibly unknown but fixed weight 

vector. The learner is shown the reward rt{At) and the vector of reward features for the chosen simple actions 
yt{At), jointly referred to as semibandit feedback. 

The goal is to achieve cumulative reward competitive with all tt € 11. Lor a policy tt, let lR(7r) := 
E(a; j, [r(7r(x))] denote its expected reward, and let tt* := argmax^gjj !k(7r) be the maximizer of 
expected reward. We measure performance of an algorithm via cumulative empirical regret, 

T 

Regret :='^rt{TT*{xt)) - rt{At). (1) 

The performance of a policy tt is measured by its expected regret, Reg(7r) := !]^(7r*) — 3^(7r). 

^Kale et al. [13] consider the favorable setting where our bounds match, when uniform exploration is valid. 
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Example 1. In personalized search, a learning system repeatedly responds to queries with rankings of search 
items. This is a contextual semibandit problem where the query and user features form the context, the simple 
actions are search items, and the composite actions are their lists. The semibandit feedback is whether the user 
clicked on each item, while the reward may be the click-based discounted cumulative gain (DCG), which is a 
weighted sum of clicks, with position-dependent weights. We want to map contexts to rankings to maximize 
DCG and achieve a low regret. 

We assume that our algorithms have access to a supervised learning oracle, also called an argmax 
oracle, denoted AMO, that can find a policy with the maximum empirical reward on any appropriate dataset. 
Specifically, given a dataset D = {xi, yi, of contexts Xi, reward feature vectors yi G with rewards 

for all simple actions, and weight vectors Vi G K^, the oracle computes 

n n L 

AMO{D) := aT:gmaxy^{vi,yi{Tr{xi))) = argmaxY^y^ Vi^£yi{Tr{xi)i), (2) 

where TT{x)e is the £th simple action that policy tt chooses on context x. The oracle is supervised as it assumes 
known features yi for all simple actions whereas we only observe them for chosen actions. This oracle is the 
structured generalization of the one considered in contextual bandits [1, 10] and can be implemented by any 
structured prediction approach such as CRTs [15] or SEARN [9]. 

Our algorithms choose composite actions by sampling from a distribution, which allows us to use 
importance weighting to construct unbiased estimates for the reward feamres y. If on round t, a composite 
action At is chosen with probability Qt{At), we construct the importance weighted feature vector yt with 
components j/t(a) := yt{a)l{a G At)/Qt{a G At), which are unbiased estimators of yt{a). For a policy tt, 
we then define empirical estimates of its reward and regret, resp., as 

^ ^ _ 

Pt{T^,w) :=-'^{w,yi{TT{xi))) and Regj(7r, w) ;= max7?t(7r', w) - 77t(7r, w). 

t ‘ ^ tt' 

i-1 

By construction, rit{n,w*) is an unbiased estimate of the expected reward lR(7r), but Reg((7r, in*) is not 
an unbiased estimate of the expected regret Reg(7r). We use [•] to denote empirical expectation over 
contexts appearing in the history of interaction H. 

Finally, we introduce projections and smoothing of distributions. For any p, G [0, l/AT] and any subdistri¬ 
bution P G A< (n), the smoothed and projected conditional subdistribution P^(A \ x) is 

I x) := (1 - Kp) Y, PWl(^(x) = A) + KpU,{A), (3) 

TT^n 

where is a uniform distribution over a certain subset of valid rankings for context x, designed to ensure 
that the probability of choosing each valid simple action is large. By mixing Ux into our action selection, we 
limit the variance of reward feature estimates y. The lower bound on the simple action probabilities under Ux 
appears in our analysis as Pniin, which is the largest number satisfying 

^ A) ^ Pminj 

for all X and all simple actions a valid for x. Note that pniin = L when there are no restrictions on the action 
space as we can take Ux to be the uniform distribution over all rankings and verify that Ux{a G A) = L/K. 
In the worst case, = 1, since we can always find one valid ranking for each valid simple action and let 
Ux be the uniform distribution over this set. Such a ranking can be found efficiently by a call to AMO for 
each simple action a, with the dataset of a single point (a;, G 1 G M^), where la(a') = l(a = a'). 
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Algorithm 1: VCEE (Variance-Constrained Explore-Exploit) Algorithm 
Require: Allowed failure probability 5 s (Ojl)- 
1: Qo = 0, the all-zeros vector. Hq = 0. Define: jit = min |l/2Ar, /5) / {Ktpram)^- 

2: for round f = 1,..., T do 

3: Let TTj.i = argmax^gn%-i(’^:'*"*) and Qt-i = Qt-i + (1 - <^^-l(7r))l7rt_l■ 

4: Observe xt S X, play At ^ I Xt) (see Eq. (3)), and observe yt{At) and rt{At). 

5: Define qt{a) = Qtl~i^ (a S A | Xt) for each a. 

6: Obtain Qt by solving OP with Ht = Ht-i U {{xt,yt{At), qt{At)} and 

7 : end for 


Semi-bandit Optimization Problem (OP) 

With history H and u > 0, define 6^ := and ib := 100. Eind Q G A<(n) such that: 

11^ II2 wMPmin — ' ^ 


Vtt G n : 


ttGII 




L 


E 


1 

Q^^{Tr{x)i G A I cc) 


< 


2KL 

Pmin 


(4) 

(5) 


3 Semibandits with known weights 

We begin with the setting where the weights w* are known, and present an efficient oracle-based algorithm 
(VCEE, see Algorithm 1) that generalizes the algorithm of Agarwal et al. [1]. 

The algorithm, before each round t, constructs a subdistribution Qt-i G A< (If), which is used to form the 
distribution Qt-i by placing the missing mass on the maximizer of empirical reward. The composite action 
for the context xt is chosen according to the smoothed distribution Qt!Ai (see Eq. (3)). The subdistribution 
Qt-i is any solution to the feasibility problem (OP), which balances exploration and exploitation via the 
constraints in Eqs. (4) and (5). Eq. (4) ensures that the distribution has low empirical regret. Simultaneously, 
Eq. (5) ensures that the variance of the reward estimates y remains sufficiently small for each policy tt, which 
helps control the deviation between empirical and expected regret, and implies that Qt-i has low expected 
regret. Eor each tt, the variance constraint is based on the empirical regret of tt, guaranteeing sufficient 
exploration amongst all good policies. 

OP can be solved efficiently using AMO and a coordinate descent procedure obtained by modifying the 
algorithm of Agarwal et al. [1]. While the full algorithm and analysis are deferred to Appendix E, several 
key differences between VCEE and the algorithm of Agarwal et al. [1] are worth highlighting. One crucial 
modification is that the variance constraint in Eq. (5) involves the marginal probabilities of the simple actions 
rather than the composite actions as would be the most obvious adaptation to our setting. This change, based 
on using the reward estimates yt for simple actions, leads to substantially lower variance of reward estimates 
for all policies and, consequently, an improved regret bound. Another important modification is the new 
mixing distribution Ux and the quantity Pmin- For structured composite action spaces, uniform exploration 
over the valid composite actions may not provide sufficient coverage of each simple action and may lead to 
dependence on the composite action space size, which is exponentially worse than when Ux is used. 

The regret guarantee for Algorithm 1 is the following: 


5 














Theorem 1. For any S € (Oj 1)> with probability at least 1 — <5, VCEE achieves re¬ 
gret 6 ( ^ L ^KT \og{N/5)/ p^n) . Moreover, VCEE can be efficiently implemented with 
0(T^/^K / (prnin log(-/V/i5))) Calls to a supervised learning oracle AMO. 

In Table 1, we compare this result to other applicable regret bounds in the most common setting, where 
w* = 1 and all rankings are valid (pmm = L). VCEE enjoys a 0{\/KLT logiV) regret bound, which is the 
best bound amongst oracle-based approaches, representing an exponentially better E-dependence over the 
purely bandit feedback variant [ 1 ] and a polynomially better T-dependence over an e-greedy scheme (see 
Theorem 3 in Appendix A). This improvement over e-greedy is also verified by our experiments. Additionally, 
our bound matches that of Kale et al. [13], who consider the harder adversarial setting but give an algorithm 
that requires an exponentially worse running time, Vl{NT), and cannot be efficiently implemented with an 
oracle. 

Other results address the non-contextual setting, where the optimal bounds for both stochastic [14] and 
adversarial [2] semibandits are Q{s/KLT). Thus, our bound may be optimal whenpmin = fl{L). However, 
these results apply even without requiring all rankings to be valid, so they improve on our bound by a s/Z 
factor when pmin = 1. This VL discrepancy may not be fundamental, but it seems unavoidable with some 
degree of uniform exploration, as in all existing contextual bandit algorithms. A promising avenue to resolve 
this gap is to extend the work of Neu [19], which gives high-probability bounds in the noncontextual setting 
without uniform exploration. 

To summarize, our regret bound is similar to existing results on combinatorial (semi)bandits but represents 
a significant improvement over existing computationally efficient approaches. 

4 Semibandits with unknown weights 

We now consider a generalization of the contextual semibandit problem with a new challenge: the weight 
vector w* is unknown. This setting is substantially more difficult than the previous one, as it is no longer clear 
how to use the semibandit feedback to optimize for the overall reward. Our result shows that the semibandit 
feedback can still be used effectively, even when the transformation is unknown. Throughout, we assume that 
the trae weight vector w* has bounded norm, i.e., ||iy *||2 < B. 

One restriction required by our analysis is the ability to play any ranking. Thus, all rankings must be valid 
in all contexts, which is a natural restriction in domains such as information retrieval and recommendation. 
The uniform distribution over all rankings is denoted U. 

We propose an algorithm that explores first and then, adaptively, switches to exploitation. In the exploration 
phase, we play rankings uniformly at random, with the goal of accumulating enough information to learn the 
weight vector w* for effective policy optimization. Exploration lasts for a variable length of time governed by 
two parameters n* and A*. The n* parameter controls the minimum number of rounds of the exploration 
phase and is 0(T^/^), similar to e-greedy style schemes [16]. The adaptivity is implemented by the A* 
parameter, which imposes a lower bound on the eigenvalues of the 2nd-moment matrix of reward features 
observed during exploration. As a result, we only transition to the exploitation phase after this matrix has 
suitably large eigenvalues. Since we make no assumptions about the reward features, there is no bound on 
how many rounds this may take. This is a departure from previous explore-first schemes, and captures the 
difficulty of learning w* when we observe the regression features only after taking an action. 

After the exploration phase of t rounds, we perform least-squares regression using the observed reward 
features and the rewards to learn an estimate w of w*. We use w and importance weighted reward features 
from the exploration phase to find a policy if with maximum empirical reward, w). The remaining 
rounds comprise the exploitation phase, where we play according to if. 
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Algorithm 2: EELS (Explore-Exploit Least Squares) 


Require: Allowed failure probability 6 G (0,1). Assume || w *||2 < B. 

Set n* ^ T2/3(A:in(Ar/5)/L)V3 niax{l, 

for f = 1,..., n* do 

Observe xt, play At ^ U (U is uniform over all rankings), observe yt{At) and rt{At). 

end for 

ut r = Et;. E,..« {v.M - 

6 : V ^ 2V + 3\n{2/6)/{2n^). 

7 : Set A* ^ max \n{iLT/S), {TV/Bf/^ (Lln(2/5))^/^|. 


8: Set E 

9 : while Aniin(S) < A* do 

10: f •(— f + 1. Observe Xt, play At ~ U, observe yt{At) and rt{At). 

11: Set S •<—E + 

12 : end while 

13 : Estimate weights w *r- yi{Ai)ri{Ai)) (Least Squares). 

14 : Optimize policy tt ^ argmax^gjj w) using importance weighted features. 
15 : Eor every remaining round; observe Xt, play At = TT{xt). 


The remaining question is how to set A*, which governs the length of the exploration phase. The ideal 
setting uses the unknown parameter V := Vara..^unif(yi) [j/(a)] of the distribution D, where Unif(yi) 

is the uniform distribution over all simple actions. We form an unbiased estimator V of V and derive an upper 
bound V. While the optimal A* depends on V, the upper bound V suffices. 

Eor this algorithm, we prove the following regret bound. 

Theorem 2. For any S G (0,1) and T > K hi{N/6) / min{L, (BL)^}, with probability at least 1 — (5, EELS 
has regret O log(At/(5))t/3 ma.-yi{B^/^L^/'^, BL^/^Y). EELS can be implemented efficiently with 

one call to the optimization oracle. 

The theorem shows that we can achieve sublinear regret without dependence on the composite action space 
size even when the weights are unknown. The only applicable alternatives from the literature are displayed in 
Table 1, specialized to B = Q{s/L). Eirst, oracle-based contextual bandits [1] achieve a better T-dependence, 
but both the regret and the number of oracle calls grow exponentially with L. Second, the deviation bound 
of Swaminathan et al. [24], which exploits the reward structure but not the semibandit feedback, leads to 
an algorithm with regret that is polynomially worse in its dependence on L and B (see Appendix B). This 
observation is consistent with non-contextual results, which show that the value of semibandit information is 
only in L factors [2]. 

Of course EELS has a sub-optimal dependence on T, although this is the best we are aware of for a compu¬ 
tationally efficient algorithm in this setting. It is an interesting open question to achieve poly (AT, L)y/T log TV 
regret with unknown weights. 


5 Proof sketches 

We next sketch the arguments for our theorems. Lull proofs are deferred to the appendices. 
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Proof of Theorem 1: The result generalizes Agarwal et. al [1], and the proof structure is similar. For the 
regret bound, we use Eq. (5) to control the deviation of the empirical reward estimates which make up the 
empirical regret Reg^. A careful inductive argument leads to the following bounds; 

Reg( 7 r) < 2 Regt( 7 r) + and Reg^( 7 r) < 2 Reg( 7 r) + cq 

Here cq is a universal constant and /i* is defined in the pseudocode. Eq. (4) guarantees low empirical regret 
when playing according to Qt**, and the above inequalities also ensure small population regret. The cumulative 

regret is bounded by ATL J2t=i which grows at the rate given in Theorem 1. The number of oracle 
calls is bounded by the analysis of the number of iterations of coordinate descent used to solve OP, via a 
potential argument similar to Agarwal et al. [1]. 

Proof of Theorem 2: We analyze the exploration and exploitation phases individually, and then opti¬ 
mize n* and A* to balance these terms. Eor the exploration phase, the expected per-round regret can be 
bounded by either Ijw or ||r(;*|j 2 V^, but the number of rounds depends on the minimum eigen¬ 

value A„iin(5i), with E defined in Steps 8 and 11. However, the expected per-round 2nd-moment matrix, 
^{x,y)~'D,A~u\y{^)y{-^Y'\ has all eigenvalues at least V. Thus, after t rounds, we expect Aniin(S) > tV, 
so exploration lasts about A*/P rounds, yielding roughly 

Exploration Regret < ^ • ||r (;*||2 minlVATP, '/l}. 

Now our choice of A* produces a benign dependence on V and yields a bound. 

For the exploitation phase, we bound the error between the empirical reward estimates 77 * (tt, w) and the 
tme reward Oi(Tr). Since we know Amin(S) > A* in this phase, we obtain 


/ K loff N 

Exploitation Regret < T|| w*|| 2 y- ^ ^ 

The first term captures the error from using the importance-weighted y vector, while the second uses a bound 
on the error \\w — w *||2 from the analysis of linear regression (assuming Amin(S) > A*). 

This high-level argument ignores several important details. First, we must show that using V instead of 
the optimal choice V in the setting of A* does not affect the regret. Secondly, since the termination condition 
for the exploration phase depends on the random variable E, we must derive a high-probability bound on the 
number of exploration rounds to control the regret. Obtaining this bound requires a careful application of the 
matrix Bernstein inequality to certify that E has large eigenvalues. 

6 Experimental Results 

Our experiments compare VCEE with existing alternatives. As VCEE generalizes the algorithm of Agarwal 
et al. [ 1 ], our experiments also provide insights into oracle-based contextual bandit approaches and this is the 
first detailed empirical study of such algorithms. The weight vector w* in our datasets was known, so we do 
not evaluate EELS. This section contains a high-level description of our experimental setup, with details on 
our implementation, baseline algorithms, and policy classes deferred to Appendix C. Software is available at 
http://github.com/akshaykr/oracle_cb. 

Data: We used two large-scale learning-to-rank datasets: MSLR [18] and all folds of the Yahoo! Learning- 
to-Rank dataset [ 6 ]. Both datasets have over 30k unique queries each with a varying number of documents 


I^ m.\n{\/KV, y/h}. 
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Dataset: MSLR 


Dataset: Yahoo! 




Number of interactions (T) 

■■■ e-Lin - VC-Lin ■■■ e-GB2 - VC-GB2 ■■■ e-GB5 - VC-GB5 - LinUCB 

Figure 1: Average reward as a function of number of interactions T for VCEE, e-GREEDY, and LinUCB on 
MSLR (left) and Yahoo (right) learning-to-rank datasets. 


that are annotated with a relevance in {0,..., 4}. Each query-document pair has a feature vector {d = 136 
for MSLR and d = 415 for Yahoo!) that we use to define our policy class. For MSLR, we choose K = 10 
documents per query and set L = 3, while for Yahoo!, we set AT = 6 and L = 2. The goal is to maximize the 
sum of relevances of shown documents (w* = 1) and the individual relevances are the semibandit feedback. 
All algorithms make a single pass over the queries. 

Algorithms: We compare VCEE, implemented with an epoch schedule for solving OP after 2*^^ rounds 
(justified by Agarwal et al. [1]), with two baselines. First is the e-CREEDY approach [16], with a constant 
but tuned e. This algorithm explores uniformly with probability e and follows the empirically best policy 
otherwise. The empirically best policy is updated with the same 2*/^ schedule. 

We also compare against a semibandit version of LinUCB [20]. This algorithm models the semibandit 
feedback as linearly related to the query-document features and learns this relationship, while selecting 
composite actions using an upper-confidence bound strategy. Specifically, the algorithm maintains a weight 
vector 6t G formed by solving a ridge regression problem with the semibandit feedback yt{at,e) as 
regression targets. At round t, the algorithm uses document features {Xa}aGA chooses the L documents 
with highest x^6t+ax^T,^^Xa value. Here, Et is the feature 2nd-moment matrix and a is a tuning parameter. 
For computational reasons, we only update Sf and 9t every 100 rounds. 

Oracle implementation: LinUCB only works with a linear policy class. VCEE and e-Greedy work 
with arbitrary classes. Here, we consider three: linear functions and depth-2 and depth-5 gradient boosted 
regression trees (abbreviated Lin, GB2 and GB5). Both GB classes use 50 trees. Precise details of how we 
instantiate the supervised learning oracle can be found in Appendix C. 

Parameter tuning: Each algorithm has a parameter governing the explore-exploit tradeoff. Eor VCEE, 
we set fit = c ^/l/KLT and tune c, in e-CREEDY we tune e, and in LinUCB we tune a. We ran each 
algorithm for 10 repetitions, for each of ten logarithmically spaced parameter values. 

Results: In Figure 1, we plot the average reward (cumulative reward up to round t divided by t) on both 
datasets. For each t, we use the parameter that achieves the best average reward across the 10 repetitions at 
that t. Thus for each t, we are showing the performance of each algorithm tuned to maximize reward over t 
rounds. We found VCEE was fairly stable to parameter tuning, so for VC-GB5 we just use one parameter 
value (c = 0.008) for all t on both datasets. We show confidence bands at twice the standard error for just 
LinUCB and VC-GB5 to simplify the plot. 

Qualitatively, both datasets reveal similar phenomena. First, when using the same policy class, V CEE 
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consistently outperforms e-GREEDY. This agrees with our theory, as VCEE achieves -\/T-type regret, while 
a tuned e-GREEDY achieves at best a rate. 

Secondly, if we use a rich policy class, VCEE can significantly improve on LinUCB, the empirical 
state-of-the-art, and one of few practical alternatives to e-GREEDY. Of course, since e-GREEDY does not 
outperform LinUCB, the tailored exploration of VCEE is critical. Thus, the combination of these two 
properties is key to improved performance on these datasets. VCEE is the only contextual semibandit 
algorithm we are aware of that performs adaptive exploration and is agnostic to the policy representation. 
Note that LinUCB is quite effective and outperforms VCEE with a linear class. One possible explanation 
for this behavior is that LinUCB, by directly modeling the reward, searches the policy space more effectively 
than VCEE, which uses an approximate oracle implementation. 


7 Discussion 

This paper develops oracle-based algorithms for contextual semibandits both with known and unknown 
weights. In both cases, our algorithms achieve the best known regret bounds for computationally efficient 
procedures. Our empirical evaluation of VCEE, clearly demonstrates the advantage of sophisticated oracle- 
based approaches over both parametric approaches and naive exploration. To our knowledge this is the first 
detailed empirical evaluation of oracle-based contextual bandit or semibandit learning. We close with some 
promising directions for future work: 

1. With known weights, can we obtain 0{^^/KLT log TV) regret even with structured action spaces? This 
may require a new contextual bandit algorithm that does not use uniform smoothing. 

2. With unknown weights, can we achieve a \/T dependence while exploiting semibandit feedback? 
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Algorithm 3: e-Greedy for Contextual Semibandits with Known Weights 

Require: Allowed failure probability 5 s (Ojl)- 
Set n = r2/3(A:in(2W/(5)/L)i/3. 

Let U be the uniform distribution over all rankings. 

For f = 1,..., n, observe xt, play At ~ U, observe yt{At) and rt{At). 

Optimize policy if ^ argmax^gjj ?7n(7'‘, w*) using importance-weighted features. 

For every remaining round: observe xt, play At = ^{xt). 


A Analysis of e-Greedy with Known Weights 


We analyze the e-greedy algorithm (Algorithm 3) in the known-weights setting when all rankings are valid, 
i.e., Pmin = L. This algorithm is different from the one we use in our experiments in that it is an explore-first 
variant, exploring for the first several rounds and then exploiting for the remainder. In our experiments, we 
use a variant where at each round we explore with probability e and exploit with probability (1 — e). This 
latter version also has the same regret bound, via an argument similar to that of Langford and Zhang [16]. 

Theorem 3. For any 6 € (0,1), when T > Kln{N/d)/L, with probability at least 1 — i5, the regret of 
Algorithm 3 is at most 0 {\\w*\\ 2T'^^^{K \og{N/ 

Proof. The proof relies on the uniform deviation bound similar to Lemma 20, which we use for the analysis 
of EELS. We first prove that for any 5 G (0,1), with probability at least 1 — <5, for all policies tt, we have 


|?7„(7r,u;*) - 3?(7r)| < 


l2K\a.{2N/5) 


n 


2K \n{2N/5) 
Ss/L n 


(6) 


This deviation bound is a consequence of Bernstein’s inequality. The quantity on the left-hand side is the 
average of n terms 

yi{TT{xi))'^w* - ¥.^^y[y{-K{x))]^w*, 

all with expectation zero, because y is unbiased. The range of each term is bounded by the Cauchy-Schwarz 
inequality as 

\\w*\\2\\m{Tt{xi)) - E^^y[y{Tr{x))]\\2 < \\w*\\2K/y/L, 

because under uniform exploration the coordinates of yi(Tt{xi)) are bounded in [0, K/L] while the coordinates 
of y{'K{x)) are in [0,1] and these are L-dimensional vectors. The variance is bounded by the second moment, 
which we bound as follows: 


^x,y,A [{y{Tr{x))'^w'*'f] < lltu^ll^IEx.y.A 


— Il'^*ll2 ^x,y,A 




. 1=1 




. 1=1 


= WfK, 


since Ex^y^A\^{'tt{x)i G A)] = L/K under uniform exploration. Plugging these bounds into Bernstein’s 
inequality gives the deviation bound of Eq. (6). 
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Now we can prove the theorem. Eq. (6) ensures that after collecting n samples, the expected reward of 
the empirical reward maximizer tt is close to max^ 3i(7r), the best achievable reward. The difference between 
these two is at most twice the right-hand side of the deviation bound. If we perform uniform exploration for n 
rounds, we are ensured that with probability at least 1 — 6 the regret is at most 

Regret < „||»*|hv^ + 2(r- n)||»*||, 

< n|k*|h^ + 3r||»*|b ^ 

For our setting of n = \n{2N/S)/L)^^^, the bound is 

4\\w*\\2T^/^{K\n{2N/d))^^^L^/^+ 3 \\w*\\2T^/^{K\ii{2N/S))‘^/^L-^^^. 

Under the assumption on T, the second term is lower order, which proves the result. □ 

B Comparisons for EELS 

In this section we do a detailed comparison of our Theorem 2 to the paper of Swaminathan et al. [24], which 
is the most directly applicable result. We use notation consistent with our paper. 

Swaminathan et al. [24] focus on off-policy evaluation in a more challenging setting where no semibandit 
feedback is provided. Specifically, in their setting, in each round, the learner observes a context a; G X, 
chooses a composite action A (as we do here) and receives reward r{A) G [—1,1]. They assume that the 
reward decomposes linearly across the action-position pairs as 

L 

E[r(A)|a;,A] = £)■ 

£=1 

With this assumption, and when exploration is done uniformly, they provide off-policy reward estimation 
bounds of the form 

This bound holds for any policy tt : X —> with probability at least 1 — <5 for any <5 G (0,1). (See 

Theorem 3 and the following discussion in Swaminathan et al. [24].) Note that this assumption generalizes 
our unknown weights setting, since we can always dehne c/>x(a, j) = w*y{a). 

To do an appropriate comparison, we hrst need to adjust the scaling of the rewards. While Swaminathan 
et al. [24] assume that rewards are bounded in [—1,1], we only assume bounded y’s and bounded noise. 
Consequently, we need to adjust their bound to incorporate this scaling. If the rewards are scaled to lie in 
[—i?, R], their bound becomes 

\Vni7r) - 3?W| < o ■ 

This deviation bound can be turned into a low-regret algorithm by exploring for the hrst n rounds, hnding 
an empirically best policy, and using that policy for the remaining T — n rounds. Optimizing the bound in n 
leads to a T^/^-style regret bound; 
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Fact 4. The approach of Swaminathan et al. [24] with rewards in [—R, i?] leads to an algorithm with regret 
bound 

O {RT‘^/^{KL\ogNf/^'^ . 

This algorithm can be applied as is to our setting, so it is worth comparing it to EELS. According to 
Theorem 2, EELS has a regret bound 

The dependence on T, K, and log N match between the two algorithms, so we are left with L and the scale 
factors B, R. This comparison is somewhat subtle and we use two different arguments. The first finds a 
conservative value for R in Eact 4 in terms of B and L. This is the regret bound one would obtain by using 
the approach of Swaminathan et al. [24] in our precise setting, ignoring the semibandit feedback, but with 
known weight-vector bound B. The second comparison finds a conservative value of B in terms of R and L. 

For the first comparison, recall that our setting makes no assumptions on the scale of the reward, except 
that the noise f is bounded in [—1,1], so our setting never admits i? < 1. If we begin with a setting of B, we 
need to conservatively set R = max{i?VL, 1}, which gives the dependence 

EELS: uiayi{B^/^L^/‘^,BL^/^) 

Swaminathan et al. [24]: max{BL®/®, 

The EELS bound is never worse than the bound in Fact 4 according to this comparison. At B = 
the two bounds are of the same order, which is 0(L^/^). For B = the EELS bound is at most 

while for B = the first term in the EELS bound is at most the first term in the Swaminathan 

et al. [24] bound. In both cases, the EELS bound is superior. Finally when B = Vt{\fL), the second term 
dominates our bound, so EELS demonstrates an improvement. 

For the second comparison, since our setting has the noise bounded in [—1,1], assume that R>1 and that 
the total reward is scaled in [—i?, i?] as in Fact 4. If we want to allow any y{A) G [0,1]^, the tightest setting 
of R is between ||w*||i/2 and ||w*||i (depending on the structure of the positive and negative coordinates 
of w*). For simplicity, assume i? is a bound on ||iy*||i. Since the EELS bound depends on B, a bound on 
the Euclidean norm of w*, we use ||w *||2 < ^ V^||w *||2 to obtain a conservative setting of B = R. 

This gives the dependence 

EELS: max{i?i/3^i/^ 

Swaminathan et al. [24]: RL^^^ 

Since i? > 1, the EELS bound is superior whenever R > Moreover, if i? = TlfsfV), i.e., at least \fL 
positions are relevant, the second term dominates our bound, and we improve by a factor of The EELS 
bound is inferior when R < which corresponds to a high-sparsity case since R is also a bound on 

||ty*||i in this comparison. 

C Implementation Details 

C.l Implementation of VCEE 

VCEE is implemented as stated in Algorithm 1 with some modifications, primarily to account for an imperfect 
oracle. OP is solved using the coordinate descent procedure described in Appendix E. 
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We set i/) = 1 in our implementation and ignore the log factor in p*. Instead, since p„im = L, we 
use pt = CyJXjKLT and tune c, which can compensate for the absence of the log(f^-/V/(5) factor. This 
additionally means that we ignore the failure probability parameter 8. Otherwise, all other parameters and 
constants are set as described in Algorithm 1 and OP. 

As mentioned in Section 6 , we implement AMO via a reduction to squared loss regression. There are 
many possibilities for this reduction. In our case, we specify a squared loss regression problem via a dataset 
D = {cci, Ai, Ui, where Xi G X, is any list of actions, yi G assigns a value to each action, and 

7 i G assigns an importance weight to each action. Since in our experiments w* = 1, we do not need to 
pass along the vectors Vi G described in Eq. (2). 

Given such a dataset D, we minimize a weighted squared loss objective over a regression class T, 

n 

/ = argmin V V - yi{a)f, (7) 

/e? fclaGA. 

where (/)(a;, a) is a feature vector associated with the given query-document pair. Note that we only include 
terms corresponding to simple actions in Ai for each i. This regression function is associated with the greedy 
policy that chooses the best valid ranking according to the sum of rewards of individual actions as predicted 
by / on the current context. 

We access this oracle with two different kinds of datasets. When we access AMO to hnd the empirically 
best policy, we only use the history of the interaction. In this case, we only regress onto the chosen actions in 
the history and we let 7 ^ be their importance weights. More formally, suppose that at round t, we observe 
context Xt, choose composite action At ~ qt and receive feedback {yt{o.t,e)}f^i- We create a single example 
{xt, At,zt,'yt) where Xt is the context. At is the chosen composite action, zt has Zt(a) = l(a G At)yt{a) 
and 7 t(a) = l/qt{a G At). Observe that when this sample is passed into Eq. (7), it leads to a different 
objective than if we regressed directly onto the importance-weighted reward features yt- 

We also create datasets to verify the variance constraint within OP. Eor this, we use the AMO in a more 
direct way by setting At to be a list of all K actions, letting yt be the importance weighted vector, and 7 * = 1. 

We use this particular implementation because leaving the importance weights inside the square loss term 
introduces additional variance, which we would like to avoid. 

The imperfect oracle introduces one issue that needs to be corrected. Since the oracle is not guaranteed to 
hnd the maximizing policy on every dataset, in the fth round of the algorithm, we may encounter a policy tt 
that has Reg(( 7 r) < 0, which can cause the coordinate descent procedure to loop indehnitely. Of course, if we 
ever hnd a policy tt with Reg^ (tt) < 0, it means that we have found a better policy, so we simply switch the 
leader. We found that with this intuitive change, the coordinate descent procedure always terminates in a few 
iterations. 


C.2 Implementation of e- Greedy 

Recall that we run a variant of e-GREEDY where at each round we explore with probability e and exploit with 
probability (1 — e), which is slightly different from the explore-hrst algorithm analyzed in Appendix A. 

Eor e-GREEDY, we also use the oracle dehned in Eq. (7). This algorithm only accesses the oracle to hnd 
the empirically best policy, and we do this in the same way as VCEE does, i.e., we only regress onto actions 
that were actually selected with importance weights encoded via 7 iS. We use all of the data, including the 
data from exploitation rounds, with importance weighting. 


14 



C.3 Implementation of LinUCB 

The semibandit version of LinUCB uses ridge regression to predict the semibandit feedback given query- 
document features a). If the feature vectors are in d dimensions, we start with Ei = Id and 9i — 0, the 
all zeros vector. At round t, we receive the query-document feature vectors a)}a^A for query Xt and 

we choose 


At = argmax < 



Since in our experiment we know that w* = 1 and all rankings are valid, the order of the documents is 


irrelevant and the best ranking consists of the top L simple actions with the largest values of the above 


“regularized score”. Here a is a parameter of the algorithm that we tune. 

After selecting a ranking, we collect the semibandit feedback {yt{at/)}i^i- The standard implementation 
would perform the update 



which is the standard online ridge regression update. For computational reasons, we only update every 
100 iterations, using all of the data. Thus, if mod(f, 100) ^ 0, we set Ej+i ^ E* and ^ Of. If 
mod(f, 100) = 0, we set 



C.4 Policy Classes 

As AMO for both VCEE and e-GREEDY, we use the default implementations of regression with various 
function classes in scikit-learn version 0.17. We instantiate scikit-learn model objects and use 
the f it {) and predict () routines. The model objects we use are 

1. sklearn.linear_model.LinearRegression() 

2. sklearn.ensemble.GradientBoostingRegressor(n_estimators=50,max_depth=2) 

3. sklearn.ensemble.GradientBoostingRegressor(n_estimators=50,max_depth=5) 

All three objects accommodate weighted least-squares objectives as required by Eq. (7). 

D Proof of Regret Bound in Theorem 1 

The proof hinges on two uniform deviation bounds, and then a careful inductive analysis of the regret using the 
OP. We only need our two deviation bounds to hold for the rounds t in which /xt = /5)/{Ktp^ya). 

Let dt '■= ln(16f^A^/(5). These rounds then start at 
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(8) 


Note that to > 4 since d* > 1 and K > Pmin- From the definition of to, we have for all t > to: 

Pt ^ \// (Ff fpmin) ; t ^ AKdt /Pmin- 


The first deviation bound shows that the variance estimates used in Eq. (5) are suitable estimators for the 
true variance of the distribution. To state this deviation bound, we need some definitions; 


V{P,TT,p) := 


E 

Vi=\ 


1 


I x) 


Vt{P,7r,p) := 


y _^_ 

Pl^{TT{x)t I X) 


(9) 


In these definitions and throughout this appendix we use the shorthand P{a \ x) to mean P{a G A\x) for any 
projected subdistribution P{A \ x). If P is a distribution, we have X]aG.A \x) = L. For a subdistribution, 
this sum can be smaller, so I 2 ;) < T for all subdistributions. The deviation bound is in the 

following theorem: 

Theorem 5. Let S G (0,1). Then with probability at least 1 — (5/8, for all t > to, all distributions P over 11, 
and all tt G 11, we have 


V{P, TT, pt) < 6AVt{P, TT, pt) + 81.3 


KL 

Pmin 


( 10 ) 


Proof. The proof of this theorem is similar to a related result of Agarwal et al. [1] (See their Lemma 10). We 
first use Freedman’s inequality (Lemma 23) to argue that for a fixed P, tt, /i, and t, the empirical version of 
the variance is close to the true variance. We then use a discretization of the set of all distributions and take a 
union bound to extend this deviation inequality to all P, tt, p, t. 

To start, we have; 

Lemma 6. For fixed P, tt, p, t and for any A G [O, ], with probability at least 1 — S: 


V(P,TT,p) - Vt{P,TT,p) < 


(e- 2)AL 


V (P, TT, p) 


ln(l/<5) 


Proof. Let: 


Zr.= Yl 


^ Pt^{TT{x^)i\Xi) 


-E, 


-D. 


E 

i=i 


Pt^{TT{x)e\x) ’ 


and notice that j '^ 1=1 = ^tiP, p) — V (P, tt, p). Clearly, EZi = 0 for all i and max^ \Zi\ < L/pp^nm 

since when we smooth by p, each simple action that tt could choose must appear with probability at least 
pPmin- By the Cauchy-Schwarz and Holder inequalities, the conditional variance is: 


< Exr„V„ f^E I — E IDLlt t \ \ \2 

\i^i ) y pt-{TT{x)i\xY 


< 


-E 




•x^D, 


E 


1 


^ Pt‘(7r(a:)^|a;) ppuTm 


V{P,TT,p). 


The lemma now follows by Freedman’s inequality. 


□ 
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To prove the variance deviation bound of Theorem 5, we next use a discretization lemma from [10] (their 
Lemma 16) which immediately implies that for any P, there exists a distribution P' supported on at most Nt 
policies such that for ct > 0, if > - 2 - 


7f MtPmin 


V{P,TT,H) - V{P',TT,fJ,t) +Ct - Vt{P,TT,Ht)^ < 7i (F (P, TT,/Tj) + Cj 14 (P, TT, ^( )) 

This is exactly the second conclusion of their Lemma 16 except we use Ct instead of their (1 + A) (we will 
set Ct > 1). The other difference is the inclusion of in the lower bound on Nt, which is based on a 
straightforward modification to their proof. 

We e=t 7, = v'sSSt + 3 ASi. ■=< = irPsiTI.= “■66K.P™n/i. The 

PtPmin 

choice of Ct is motivated by Lemma 6, which can be rearranged to (for a distribution P') 

1 . . 1 ln(l/i)) 


V{P ,7r, fit) _ (e-2)LAt ^ (e-2)LAt 

/TfPmin /TfPmin 


1 - 


^V{P' ,TT,flt) - CtVt{P' ,'IT,flt) < Ct 


ln(l/^) 

tXt 


To take a union over all f S N, -point distributions P over If, and all tt € If, we set 5t = ] 

the fth iteration. This inequality becomes 

ln(2Af^‘+ifV<5) 


V{P' ,-K,flt) - CtVt{P' ,'K,fJLt) < Cf 


tXt 


The choice of Ct and Xt leads to a bound C( = 


l-0.66(e-2) — 

We also use the values of Nt and jt to bound 


< 1.91. 


It = 


1 - Kfit ^ 1 - Kfit 
NtfttPmin 


, 1 1 

< \ -h -. 

12 4 


NtPtPn 

Rearranging the discretization claim gives 

V{P,7r,fit) < J^^^yt{P,7T,fit) + (v{P',7T,fit) - CtVt{P',7T,fit)) 

Ct \n{2N^^+H^/d) 


(1 - 7i) 

< 6.414(P,7r, pt) -f 


(1 - jt) tXt 

Using the bounds on ct, 7 * and the settings of Nt and At, this last term is at most 
Ct 


(1 - 7t) 


L\tl{2NH'^/5) , LiVt In(At) ^ 6.3Pln(16iV2f2/5) ^ 75L(1 - PTpt) In(iV) 


ifitPmin 


tptPn 


PttPa 


,2+.jj2 

min 


KtPn 


The theorem now follows from the bounds of Eq. (8). 


□ 


The other main deviation bound is a straightforward application of Freedman’s inequality and a union 
bound. To state the lemma, we must introduce one more definition. Let 

Vt(7r) := max V{Qr,TT,fir) 

0<T<t—l 

where Qt is the distribution calculated in Step 3 of Algorithm 1 . Note that is the distribution used to 
select the composite action in round r -f 1. 
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Lemma 7. Let 5 € (0,1). Then with probability at least 1 — 5/A, for all t >tQ and w € 11, we have 

(11) 

Proof. Consider a specific t > to and tt S 11. Let 


Zi := {w*,yi{Tr{xi))) - {w*,yi{Tr{xi))) 

and note that ^ — 3?(7r). Since y^ is an unbiased estimate of the Z^s form a 

martingale. The range of each Zi is bounded as 


l^.l < lk*l!illy. 


Z/z II oo 





< 


Ik^^lll 

ftiPmin 


because the y^s are non-increasing. The conditional variance can be bounded via the Cauchy-Schwarz 
inequality: 




P(7r(a;)^ 


£=1 


Q,ll"{T^ix)l I x) 


< \\w*\\lV{Q,_i,TT,yi_i) < 




By Freedman’s inequality with A = p,tPmm/\\'w*\\i, we have, with probability at least 1 — S/{ 8 t'^N), 


\VtiTr,w*) - 3l(7r)| < 


PtPn 


< 


< 


w 111 




w” 1 


t PtPn 


W 111 


* Vt(7r)pminPt + Kpt\\w* 


W 111 


-Vt(7r)pminPt 


"*'li 

\w*\\l 


KLyit. 


( 12 ) 

(13) 


Here, Eq. (12) follows because dt/{pmmt) < Kpf by Eq. (8). Eq. (13) follows because ||ru*||i < 
L||w*|||/||rt;*|ii by the fact that ||iu*||i < '/L\\w*\\ 2 . The lemma follows by a union bound over all 
t >to and TT G n. □ 


Equipped with these two deviation bounds we will proceed to prove the main theorem. Eet £ denote 
the event that both the variance and reward deviation bounds of Theorem 5 and Lemma 7 hold. Note that 
E(£) > 1 — S/2. Using the variance constraint, it is straightforward to prove the following lemma: 

Lemma 8. Assume event £ holds, then for any round t > 1 and any policy tt G If, let f* be the round 
achieving the max in the definition ofVt{'!T). Then there are universal constants 9i > 2 and 62 such that: 


2KL 


i Pmm 

9iKL ^ llrulli Regt,(7r) 

Pmin \\w*\\l 6»2PmmFi» 


ifU < to. 


ifU > to. 


(14) 
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Proof. The first claim follows by the definition of Vt( 7 r) and the fact that = 1/2K for r < For 
the second claim, we use the variance deviation bound and the optimization constraint. In particular, since 
t* > toi we can apply Theorem 5: 

KL 

V{Qt,,Tr,p,tJ < 6.4yt^(Qt^,7r,p4j + 81.3-, 

Pmin 


and we can use the optimization constraint which gives an upper bound on Vt^ (Qt*, tt, 

< VtAQt.,^,Pu) < — + 

The bound follows by the choice 9i = 94.1 and 82 = i/'/S-d. 

We next compare Reg( 7 r) and Reg( 7 r) using the variance bounds above. 

Lemma 9. Assume event £ holds and define cq := 4(1 + 61 ). For all t > Iq and all policies tt S II.' 


□ 


Reg( 7 r) < 2 Regj( 7 r) +co| 






-KLpt and Regj( 7 r) < 2 Reg( 7 r) + cq 




llw’lli 


■-KLpf 


(15) 


Proof The proof is by induction on t. As the base case, consider t = tg where we have p.T = 1/ {2K) for all 
T < to, SO Vi(7r) < 2iTL/pmin for all TT e n by Lemma 8 . Using the reward deviation bound of Lemma 7, 
which holds under £, we thus have 


- 3?(7r)| < tQ^VtiTT)pM + l^KLpt < 


iw-lli 




Iw’lli 


for all TT G n. Now both directions of the bound follow from the triangle inequality and the optimality of tt; 
for pt (•) and tt* for 3?(-), using the fact that Cq > 6 from the dehnition of di. 

For the inductive step, hx some round t and assume that the claim holds for all to < t' < t and all tt G 11. 
By the optimality of for rjt and Lemma 7, we have 

Reg( 7 r) - Regj( 7 r) = ( 3 ?( 7 r*) - 3 l( 7 r)) - w*) - ptA, w*)) 

< ( 3 l( 7 r*) - 3 l( 7 r)) - w*) - 774 ( 71 , w*)) 




1 




Now by Lemma 8 , there exist rounds i,j < t such that 


V I Ill'll 1 ^^ 

Vt( 7 r) <-h II ^112 - 1 (* > to) 

Pmin ||tU 112 92Pmmd'i 


^ 4 ( 71 *) < 


,KL 


||y;^||i Reg^-(7r^) 


Pmin 11^*112 ^2PmmP'j 

For the term involving ^ 4 ( 77 ), if f < to, we immediately have the bound 


l(j > to) 
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On the other hand, if i > to using the fact that fii > fit ^ and applying the inductive hypothesis to Reg^(7r) 
gives: 






Pminf^t ^ 


klli 


KL^t + 


Reg,(7r)/xt 


Similarly for the Vt{Tr^,) term, we have the bound 


Vt(7r*) 




*||2 


PminPt ^ 




,*||2 


since tt* has no regret. Combining these bounds gives: 


Reg(7r) - Regj(7r) < 2 ( 6»i + ^ 

P2 


„*l|2 


< 


02 


■KL^it 


2Reg(7r*) 


02 


-KL^t + 


2Reg(7 


\w*\\lr^r , 2Reg(7r) 

\w*h 02 


= ( 01 + ^ 




*||2 


-KL^t, 


„*l|2 


+ 2 


-KLfit, 


which gives 

^ TirW +1) M^ 

Recall that 0i = 94.1,02 = i/)/6.4, t/> = 100, and cq = 4(1 + 0i). This means that 02 > 15.6, so 2/02 < 1/2, 
and hence the pre-multiplier on the Reg((7r) term is at most 2. To finish proving the bound on Reg(7r), it 
remains to show that cq > 2(1 + 0i -f co/ 02 )/(l — 2 / 02 ), or equivalently, that 


co(1-4/02)>2(1 + 0i). 

This holds, because co(l — 4/02) = 4(1 — 4/02)(1 -I- 0i) and 4/02 < 1/2. 

The other direction proceeds similarly. Under event £ we have: 

Reg((7r) - Reg(7r) = r]t{Trt,w*) - rjtiir, w*) - Dl(7r*) -f 3^(7r) 
< TjtiTTt, w*) - ?7t(7r, w*) - 3l{7Tt) + 3?(7r) 


< +"^t(7rt)) 




I Pminpt 4“ 2 II 

1 k* 


■KLut. 


As before, we have the bound: 

Vt(7r) 


kill 




PmmPt ^ 






02 


but for the Vt{TTt) term we must use the inductive hypothesis twice. We know there exists a round j < t for 
which 


vuxf) 

- kmm |k1li02PmmM7 


l(j > to). 


Applying the inductive hypothesis twice gives: 


\w*\\i ||u;*||i (2Reg(^i) + co^iTL/Xj 


W*\\l 02PminMj Ik 


- Il,„*l|2 


^ ll/i( 

- Il,„*l|2 


W’' 


02Pmin/rj 

^ 2 (2R^,(7rt) + + co{ 

02Pmin/7j 


-KL^j 


< 


3co KL 

02 Pmin 
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Here we use the inductive hypothesis twice, once at round j and once at round t, and then use the fact that 
has no regret at round t, i.e., Reg(( 7 rt) = 0. We also use the fact that the fits are non-increasing, so fit/fJ-j < 1- 
This gives the bound; 




w* 1 


< I KPA-L^., 


6 ». 


1 


Combining the bounds for Vt( 7 r) and Vt{Trt) gives: 


Regt( 7 r) < ( 1 -h ^ ) Reg( 7 r) -f ( 26»i -f ^ -f 2 


w* 1 


■-KLfit. 


Since 6*2 > 2, the pre-multiplier on the first term is at most 2. It remains to show that cq > 2(1-|- 4 co/ 02 - 
This is again equivalent to co(l — 4 / 6 * 2 ) > 2(1 + 6 * 1 ), which holds as before. □ 


The last key ingredient of the proof is the following lemma, which shows that the low-regret constraint in 
Eq. (4), based on the regret estimates, actually ensures low regret. 

Lemma 10. Assume event £ holds. Then for every round t > 1." 

^ Qt_i( 7 r)Reg( 7 r) < {Ai/ + cq) ||^ ||^ (16) 


Proof. If f < fo then fit-i = 1/{2K) in which case (since Reg( 7 r) < ||r(;*||i): 

i:0._.WRegW < ||»*|1. < j//|f i < (4g, + c„)t/iALf,-.. 

For t > to, we have; 


Q4_i(7r)Reg(7r) < Y <9t-iW ( 2Regt_i(7r) -f 




*\\2 


coi 


TT^n 


ttGII 




■-KLfit-1 


< 1^2^ Qt_i(7r)Regj_i(7r)j + co'jf^^KLfit-i 


\w 


.★||2 


< (4V; + co) KLfit-i. 

Ik*lii 


The first inequality follows by Lemma 9 and the second follows from the fact that Qt-i places its remaining 
mass (compared with Qt-i) on irt-i which suffers no empirical regret at round f — 1. The last inequality is 
due to the low-regret constraint in the optimization. □ 

To control the regret, we must first add up the fitS, which relate to the exploration probability: 

Lemma 11. For any T > 1: 


T 

Y, pt-i < 2 

t=i 
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Proof. We will use the identity 

i< ^ "’> 

K y APmin 

which holds, because dr > 1 and K > Pmin- We prove the lemma separately for T = 1 and T > 2. Since 
to > 4, we have /ig = 1/2K. Thus, for T = 1, by Eq. (17): 


= — < -\l— < 2t/^ 

^ 2K - 2\l Kp^i, - Y 


For T > 2, we use the fact that //q = /^i = and fit < ^/dr /{Ktpmm) for t <T: 

T , I ; T 


E^*-i - 


+ 


Kp, 


-E 

™ t=3 


1 


< 


' dx 
Kpmi 


+ 




(18) 


< 2, 


rd7 




In Eq. (18), we bounded the first term using Eq. (17) and the second term using the telescoping identity 
1 / ^/t — 1 < 2\Jt — 1 — 2 y/t — 2, which holds for t > 2. □ 

We are hnally ready to prove the theorem by adding up the total regret for the algorithm. 

Lemma 12. For any T G N, with probability at least 1 — (5, the regret after T rounds is at most: 


w* 1 


2y'2T\n{2/5) + 2(4^: + cg + 1). 


iKTdj 

Pmm 


Proof For each round t > 1, let Zt ■■= rt(7r*(xt)) - rt{At) - Qt-i Since at round t, 

we play action At with probability Qtl~f (At), we have EZt = 0. Moreover, since the noise term ^ is shared 
between rt{^T^,(xt)) and rt{At), we have \Zi\ < 2||u>*|ji and it follows by Azuma’s inequality (Lemma 24) 
that with probability at least 1 — S/2: 


T 

Y,\Zt\<2\\w*\\W2TH2/5). 

t=i 

To control the mean, we use event £, which, by Theorem 5 and Lemma 7, holds with probability at least 


22 






























Algorithm 4: Coordinate Ascent Algorithm for Semi-Bandit Optimization Problem (OP) 

Require: History H and smoothing parameter /i. 

1 : Initialize weights Q ^ 0 G A<(n). 

2: while true do 
3: For all tt, define; 


14 (Q) := 



1 

Qt^{TT{x)i\x) 


D^(Q) := 14 (Q) 


2KL 

b-TT 

Pmin 


Sn(Q) '■= 


E 


^ Qt^(TT{x)e\x)‘^ 


4: 

5: 

6 : 

7: 


If replace Q by cQ where c ^ 


2KL/pr„ 


< 1 . 


- QM{2KL/p„i„+b^ 

Else if 3-77 s.t. D^{Q) > 0, update Q{t:) ^ Q{t:) + a-„{Q) where a-^{Q) := 2 {i-itJ)s}^Q) ■ 

Otherwise halt and output Q. 

end while 


1 — (5/2. By another union bound, with probability at least 1 — (5, the regret of the algorithm is bounded by: 

T 

Regret < 2\\w*\\i^y2T\n{2/6) + 

t—1 TT^n 
T 

< 2||w*||iV2Tln(2/(5) + EE 1^(1 - Knt-i)Qt-i{n)Reg{Tr) + ||w*|| i AT/xt-ij 


t—1 TT^n 

^ '1^*112 


< 2||u;*|| 1 ^2T\n{2/S) + V(4V7 + co + 1) 

Ik 111 


< 


\w 


*||2 


Iw’lli 


2k2rin(2/<5) + 2(4^7 + Cq + 1). 


I KTdr 

Pmin 


Here the first inequality is from the application of Azuma’s inequality above. The second one uses the defini¬ 
tion of Qkk rounds where we play as Qt-i and rounds where we explore. The exploration rounds 

occur with probability K^t-i, and on those rounds we suffer regret at most |k*||i. For the other rounds, we 
use Lemma 10 and then Lemma 11. We collect terms using the inequality ||ui*|ji < L||u>*|| 2 /||ui*|| i. □ 


E Proof of Oracle Complexity Bound in Theorem 1 

In this section we prove the oracle complexity bound in Theorem 1 . First we describe how the optimization 
problem OP can be solved via a coordinate ascent procedure. Similar to the previous appendix, we use the 
shorthand Q{a \ x) to mean Q{a G A | x) for any projected subdistribution Q{A \ x). If Q is a distribution, 
we have \ x) = L. For a subdistribution, this number can be smaller. 

This problem is similar to the one used by Agarwal et al. [ 1 ] for contextual bandits rather than semibandits, 
and following their approach, we provide a coordinate ascent procedure in the policy space (see Algorithm 4). 
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There are two types of updates in the algorithm. If the weights Q are too large or the regret constraint in 
Equation 4 is violated, the algorithm multiplicatively shrinks all of the weights. Otherwise, if there is a policy 
that is found to violate the variance constraint in Equation 5, the algorithm adds weight to that policy, so that 
the constraint is no longer violated. 

First, if the algorithm halts, then both of the conditions must be satisfied. The regret condition must 
be satisfied since we know that Q{'k){2KL /+ ^tt) < 2iTL/pniin which in particular implies that 

— 2iTT/pmin as required. Note that this also ensures that < 1 so Q € ^<(n). 

Finally, if we halted, then for each tt, we must have D,r(Q) < 0 which implies 14-(Q) < so the 

variance constraint is also satisfied. 

The algorithm can be implemented by first accessing the oracle on the importance weighted history 
{(xr, yr, w*)}t=i at the end of round t to obtain tt^, which we also use to compute The low regret check 
in Step 4 of Algorithm 4 can be done efficiently, since each policy in the support of the current distribution Q 
was added at a previous iteration of Algorithm 4, and we can store the regret of the policy at that time for no 
extra computational burden. This allows us to always maintain the expected regret of the current distribution 
Q for no added cost. Finding a policy violating the variance check can be done by one call to the AMO. At 
round t, we create a dataset of the form {xi, Zi,Vi) of size 2t. The first t terms come from the variance 14 (Q) 
and the second t terms come from the rescaled empirical regret 6^. For t < t, we define Xr to be the r* 
context. 


Zr{a) 


1 

tQf^{a\xr) ’ 


and Vr '■= i- 


With this definition, it is easily seen that 14 (Q) = Y^t=i For t > t, we define Xr to be the 

context from round t — t and 


Zr{a) 




Vria), 


and Vr '■= w*. 


It can now be verified that X)r*=t+i recovers the 6^ term up to additive constants independent of the 
policy TT (essentially up to the ?7t(7rt) term). Combining everything, it can be checked that: 


2t 

('^'T )) ) t^r) 

T -1 


2KL 

Pmin 


Ik^lll VtjTTt) 

Ik* II 2 


The two terms at the end are independent of tt so by calling the argmax oracle with this 2t sized dataset, we 
can find the policy tt with the largest value of If the largest value is non-positive, then no constraint 
violation exists. If it is strictly positive, then we have found a constraint violator that we use to update the 
probability distribution. 

As for the iteration complexity, we prove the following theorem. 

Theorem 13. For any history H and parameter p. Algorithm 4 halts and outputs a set of weights Q S A< (If) 
that is feasible for OP. Moreover, Algorithm 4 halts in no more than iterations and each iteration 

can be implemented efficiently, with at most one call to AMO. 

Equipped with this theorem, it is easy to see that the total number of calls to the AMO over the course of 
the execution of Algorithm 1 can be bounded as O setting of pf Moreover, due 

to the nature of the coordinate ascent algorithm, the weight vector Q remains sparse, so we can manipulate 
it efficiently and avoid running time that is linear in N. As mentioned, this contrasts with the exponential- 
weights style algorithm of Kale et al. [13] which maintains a dense weight vector over A<(n). 
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We mention in passing that Agarwal et al. [ 1 ] also develop two improvements that lead to a more efficient 
algorithm. They partition the learning process into epochs and only solve OP once every epoch, rather than in 
every round as we do here (Lemma 2 in Agarwal et al. [1]). They also show how to use the weight vector 
from the previous round to warm-start the next coordinate ascent execution (Lemma 3 in Agarwal et al. [1]). 
Both of these optimizations can also be implemented here, and we expect they will reduce the total number 
of oracle calls over T rounds to scale with v/T rather than as in our result. We omit these details to 
simplify the presentation. 

E.l Proof of Theorem 13 

Throughout the proof we write U{A \ x) instead of Ux{A) to parallel the notation Q{A \ x). Also, similarly 
to Q{a I x), we write U{a\x) to mean Ux{a G A). 

We use the following potential function for the analysis, which is adapted from Agarwal et al. [1], 

Ex^h [RE(C/(- I x) II Q^(- I x))] 

+ 2K/p^, 

with 


RE(p|lg) := ^Paln(Pa/<?a) +ga -Pa 

CL^J\. 


being the unnormalized relative entropy. Its arguments p and q can be any non-negative vectors in K^. For 
intuition, note that the partial derivative of the potential function with respect to a coordinate Q{'k) relates to 
the variance I4-(Q) as follows; 


9<i)((5) Z]QG7r(a;) ( Qi(i|i) ^p) + {I Kp)'j 


(9(5 (tt) 


= -E 


Xr^H 


E 


1 - Kp 
U{a\x) 


Q>^ia I x) 


L 


Pminb 7T 

2K 


<~VAQ) + L + Pfp^ 

- EaA f_2V (O) + -b b 

= ^{-DAQ)-vAQ))- 


b-K 


2K/p„ 


This means that if Dt^^Q) > 0, then the partial derivative is very negative, and by increasing the weight Q{7r), 
we can decrease the potential function $. 

We establish the following five facts: 

1. m<LHl/{Kp))/{l-Kp). 

2. $((5) is convex in Q. 

3. $((5) > 0 for all Q. 
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4. The shrinking update, when the regret constraint is violated, does not increase the potential. More 
formally, for any c < 1, we have ^(cQ) < ^(Q) whenever Q{'k){2KL/p^i^ + 6^) > 2KL/p^i^. 

5. The additive update, when > 0 for some tt, lowers the potential by at least 

With these hve facts, establishing the result is straightforward. In every iteration, we either terminate, perform 
the shrinking update, or the additive update. However, we will never perform the shrinking update in two 
consecutive iterations, since our choice of c, ensures the condition is not satished in the next iteration. Thus, 
we perform the additive update at least once every two iterations. If we perform I iterations, by the hfth fact, 
we are guaranteed to decrease the potential T* by, 

2 4:{1-Kp) ~ 8{1-Kp) 

However, the total change in potential is bounded by L \n{l/{Kp,))/ (1 — Kp) by the hrst and second facts. 
Thus, we must have 


ILfJ.Pmin ^ L\n{l/{Kp)) 
8{l-Kp)- (l-Kp) 


which is precisely the claim. 

We now turn to proving the five facts. The first three are fairly straightforward and the last two follow 
from analogous claims as in Agarwal et al. [1]. To prove the hrst fact, note that the exploration distribution in 
<5^ is exactly Ux, so 


Uia\x) -il-Kp)Uia\x)^ ^ Lln{l/{Kp)) 

l-Kp - l-Kp ' 

because U{a\ x) = L since 17(A | a;) is a distribution. Convexity of this function follows from the 

fact that the unnormalized relative entropy is convex in the second argument, and the fact that the weight 
vector q G with components qa = Q^{a | a;) is a linear transformation of Q S K^. The third fact follows 
by the non-negativity of both the empirical regret and the unnormalized relative entropy RE(-|j •). For the 
fourth fact, we prove the following lemma. 


$( 0 ) = 


E 

a£A 


Lemma 14. Let Q be a weight vector for which '^^Q{'K){2KL/px{m + ^-n-) > ‘^KL/p^i^ cmd define 
^ ■= E. Q(^"k2kE/p„.„+6.) < 1- < ^(0)- 

Proof. Let g[c) := $(cQ) and Q>^{a\x) := (1 — Kp)cQ{a\x) + KpU{a\x). By the chain rule, using the 
calculation of the derivative above, we have: 


ff'(c) 




d^{cQ) 

dQ{-K) 


Pmm 

2K 


E'^w 


2KL 

Pmin 



-^gWE, 


E 

aG'7r(ai) 


U{a\x) 

Qc{a\x) 


(19) 
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Analyze the last term; 


^Q(7r)E, 


U (ct|x) 

ifr 

U{a\x)Q{Tr)l{a G 7r(x)) 

^ Q^ia\x) 

a^7T{x) 


^ ^ Qc(alx) 

laeA-ireU 1 , j 


= E, 


E 

.a^A. 


U {a\x)Q{a\x) 
Qc{a\x) 


= -E, 

c 


E 

,cl^A 


U {a\x)cQ{a\x) 
Qcia\x) 


( 20 ) 


We now focus on one context x and define qa ■= cQ{a\x) and Ua '■= U{a\x)/L. Note that U{a\x) = L 
so the vector u describes a probability distribution over a G A. The inner sum in Eq. (20) can be upper 
bounded by; 


E U{a\x)cQ{a\x) _ _ LUaQa 

Qc{a\x) ^ - 


aGA 


= E7T3- 


LUaiqa/Ua) 


(1 - Kfl)qa + KL^Ua ^ (1 - Kfi)(qa/Ua) + KLfj, 


= LEn. 


qa/Ua 


< 


(1 - Kfl){qa/Ua) + KLfl 


< 


(1 - Kfl)Ear^u[qa/Ua\ + KL^l 
_ ^(^a^A 9a) _ 

(1 - Kfi){Y,aeA la) + KLu 


{I- K^)L + KLfi 


( 21 ) 


In the third line we use Jensen’s inequality, noting that x({ax + b) is concave in a; for a > 0. In Eq. (21), we 
use that qa ^ L and that xjiax + b) is non-decreasing, so plugging in L for 9a gives an upper 

bound. 

-, we obtain 


2KL/pra 


Combining Eqs. (19), (20), and (21), and plugging in our choice of c = ^ Q{Tr){ 2 KL/p +b 
the following lower bound on g'{c): 




l'2KL 


Pmin 


+ b-jr ] - 


Pmin 


Eq(" 


f2KL 


V Pmin 


+ b^r ] — 


2Kl\ 

CPmin J 


= 0 . 


Since g is convex, this means that g{c') is nondecreasing for all values c' exceeding c. Since c < 1, we 
have; 


^(Q) = 9(1) > 9(c) = ^{cQ). 


□ 


And for the fifth fact, we have; 

Lemma 15. Let Q be a subdistribution and suppose, for some policy tt, that Dt^{Q) > 0. Let Q' be the new 
set of weights which is identical except that Q' {it) ;= Q(tt) + a with a ;= a,r(Q) > 0. Then 


‘i>(Q) - m') > 


Lgpn 


4(1-ATp)’ 
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Proof. Assume DT^iCf) > 0. Note that the updated subdistribution equals Q'{-) = Q{-) + al(- = tt), so its 
smoothed projection, Q'^{a \ x) = Q^{a | a;) + (1 — Kix)a\{a G 7r(x)), differs only in a small number of 
coordinates from Q^{a \ x). Using the shorthand := Q^{a \ x), q'f' := Q'^{a \ x) and Ua ■= U{a\ x), we 
have; 


2K($(Q) - $(Q')) = 2K 


E, 


Ea {ua HUa/q^) - Ua ln{Ua/qT) + Qa - QT 


ab^. 


2K 


-E, 


1 - Kn 


> E, 

- I-K^i " 




(1 - K^i) 

2KcxL rr^TrPmin 


2K/p„ 


Qa 




2 G 7 t{x 


■ Pmin^ 


/2KL 

\ Pmin 


+ bj, 


The term inside the expectation can be bounded using the fact that ln(l + x) > x — x^/2 for x > 0; 

V 1 fl I ^T& (a{l-Kp) a‘^{l-Kpf' 

^ \ ^ Q^(a|x) ) 


E„ 


aG7r(x) 


> E., 


E 

2G7r(a:) 


Qu[a I x) 2Q^(a | x)^ 


= a(l - Kp)V^iQ) - E^^£i£5^(Q). 


Plugging this in the previous derivation gives a lower bound; 


2K{^{Q) - $(Q')) > 2pmmQ!l4((5) - (1 - KSt,{Q) -Ptnino; 

\ Pmin 

> Prmna{V^{Q) + D^{Q)) - (1 - Kp)p„^i„a'^S^{Q), 

V^{Q)+D^iQ) wpnhtoin- 
2(i-Kfj.)s^{Q) ’ Obtain. 

Pmin(U,(Q)+U>.(Q))" 


67 T 


using the definition Dt^{Q) = K-CQ) — Since a = 


2iT($(Q) - $(Q')) > 


^{l-Kp)S^{Q) 


Note that S^^iQ) > (by bounding the square terms in the definition of S',r((5) by a linear term 

times the lower bound, which is ppmm) and that 14(Q) > since Dt^{Q) > 0. Therefore; 


2i^($(Q) - $(Q')) > 


PpLaiV^Q) + D^{Q)f 

4(1 - Kp)V^{Q) 


> ppL^y^iQ) 

- A{l-Kp) 


> 


K L pPaiin 
2(1 - Kp)' 


Dividing both sides of this inequality by 2K proves the lemma. 


□ 


F Proof of Theorem 2 

The proof of Theorem 2 requires many delicate steps, so we first sketch the overall proof architecture. 
The first step is to derive a parameter estimation bound for learning in linear models. This is a somewhat 
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standard argument from linear regression analysis, and the important component is that the bound involves the 
2nd-moment matrix S of the feature vectors used in the problem. Combining this with importance weighting 
on the reward features y as in VCEE, we prove that the policy used in the exploitation phase has low expected 
regret, provided that E has large eigenvalues. 

The next step involves a precise characterization of the mean and deviation of the 2nd-moment matrix E, 
which relies on the exploration phase employing a uniform exploration strategy. This step involves a careful 
application of the matrix Bernstein inequality (Lemma 26). We then bound the expected regret accumulated 
during the exploration phase; we show, somewhat surprisingly, that the expected regret can be related to the 
mean of 2nd-moment matrix E of the reward features. Finally, since per-round exploitation regret improves 
with a larger setting A*, while the cumulative exploration regret improves with a smaller setting A*, we 
optimize this parameter to balance the two terms. Similarly, the per-round exploitation regret improves with a 
larger setting n*, while the cumulative exploration regret improves with a smaller setting n*, and our choice 
of n* optimizes this tradeoff. 

An important definition that will appear throughout the analysis is the expected reward variance, when a 
single action is chosen uniformly at random: 






fiG*A 




( 22 ) 


F.l Estimating V 

The first step is a deviation bound for estimating V. 

Lemma 16. After rounds, the estimate V satisfies, with probability at least 1 — 5, 




IV\n{2/5) ^ ln(2/(5) 
n* 6n* 


Proof. Note that our estimator, V = :^ X]”=i is an average of i.i.d. terms, with 

1 

2 

a^h^A. 


Zt := 




where (7 is a uniform distribution over all rankings. The mean of this random variable is precisely V: 


'^(x,y)^'D,A^u[Zt] — 2^2 










2^ E - Ma)yib) + 


a,,bGA 








= V. 
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Since we choose L actions uniformly at random, the probability for two distinct actions jointly being selected 
is U{a,b & A) = and for a single action it is U{a G A) = LjK. The {y{a) — y{b))‘^ term is at most 

one but it is always zero for a = b, so the range of Zt is at most 


0<Zt< 


— y 

2iT2 

a^b^At 


K{K- 1) 


K{K -1) ^ 1 
^ 2 ' 


Note that the last summation is only over the L{L — 1) action pairs corresponding to the slate At, as the 
indicator in Zt eliminates the other terms in the sum over all actions from A. 

As for the second moment, since Zt G [0,1/2], we have 

E[Z/] < E[Zt]/2 < V/2. 


By Bernstein’s inequality, we are guaranteed that with probability at least 1 — <5, after n* rounds. 




V\n{2/5) 


n* 


ln(2/(j) 

6n* 


□ 


Equipped with the deviation bound we can complete the square to find that 
' 51n(2/,5) 




4n* 


12n* 
ln(2/(j) 


< 1 >+ „ 

y 4n* I 2n- 


V < 


hn(2/S) 


4n* 




<2V + 


31n(2/,5) 

2n* 


Our definition of A* uses V which is precisely this final upper bound. Working from the other side of the 
deviation bound, we know that 


V 4n* I 2n* 


And combining the two, we see that 




51n(2/b) 

2n* 


(23) 


with probability at least 1 — (5. 


F.2 Parameter Estimation in Linear Regression 

To control the regret associated with the exploitation rounds, we also need to bound || w — w *||2 which follows 
from a standard analysis of linear regression. 
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At each round t, we solve a least squares problem with features yt{At) and response rt which we know 
has E[rt | yt, At\ = yt{At)'^w*. The estimator is 

t 

wt := argmin^ {yi{Ai)'^w - n) . 

W . . 

2=1 

Define the 2nd-moment matrix of reward features, 

t 

■='^yi{A^)yi{A,)'^, 
i=l 

which governs the estimation error of the least squares solution as we show in the next lemma. 

Lemma 17. Let denote the 2nd-moment reward matrix after t rounds of interaction and let wt be the 
least-squares solution. There is a universal constant c > 0 such that for any 5 S (0, 2/e), with probability at 
least 1 — (5, 

Ikt - < cL\tl(2/5). 

Proof This lemma is the standard analysis of fixed-design linear regression with bounded noise. By definition 
of the ordinary least squares estimator, we have TnWt = where Yi-t G is the matrix of features, 

ri-t G M* is the vector of responses and = Yj^^Yi-t is the 2nd-moment matrix of reward features defined 
above. The true weight vector satisfies Ytw* = Y^^{ri.t — where S is the noise. Thus 
Yt{wt — w*) = and therefore, 

Ikt - = {wt - w*)'^Yt{wt - w*) = (wt - w*)'^YtT,lYtiwt - w*) = ^l^Yl,tT,lY^^^l.,t, 

where is the pseudoinverse of Et and we use the fact that AA'^A = A for any symmetric matrix A. Since 
s| = the matrix is a proj ection matrix, and it can be written as where U G M* ^ ^ 

is a matrix with L' orthonormal columns where L' < L. We now have to bound the term = 

Let Hxy = (xi, i/i ,... ,Xt, yt) denote the history excluding the noise. Conditioned on H^y, 
the vector is a subgaussian random vector with independent components, so we can apply subgaussian 
tail bounds. Applying Lemma 25, due to Rudelson and Vershynin [23], we see that with probability at least 

l-<5, 

^ltUU^^i:t < I H^y] + ^Jco\\UUT\\j,ln{2/6) + co\\UU^\\ ln{2/6) (24) 

< (^L \J coLln(2/b) -f cq ln(2/<5)^ < {y/L -|- \J cq ln(2/<5)^ . 

To derive the second line, we use the fact that UU'^ is a projection matrix for an L'-dimensional subspace, so 
its Frobenius norm is bounded as ||C/C/^|||. = tr(f7C/^) = L' < L, while its spectral norm is ||f7f7^|| = 1. 
The expectation in Eq. (24) is bounded using the conditional independence of the noise and the fact that its 
conditional expectation is zero: 

E[^'[.y.UU'^^i:t I H^y] = tT(uu'^E[^i.,t^f.^ I H^y]^ = trdiug(E| Xi,y^])i<t^ 

< tr((7(7^) ^maxE[^/ | Xi,yi]j < L' < L. 

Finally, when 6 G (0,2/e) and with c = (1 + -fco)^, we obtain the desired bound. □ 
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F.3 Analysis of the 2nd-Moment Matrix S* 

We now show that the 2nd-moment matrix of reward features has large eigenvalues. This lets us translate 
the error in Lemma 17 to the Euclidean norm, which will play a role in bounding the exploitation regret. 
Interestingly, the lower bound on the eigenvalues is related to the exploration regret, so we can explore until 
the eigenvalues are large, without incurring too much regret. 

To prove the bound, we use a full sequence of exploration data, which enables us to bypass the data- 
dependent stopping time. Let {xt, yt, At,^t}f=i be a sequence of random variables where {x, y,^) ^ D and 
At is drawn uniformly at random. Let wt be the least squares solution on the data in this sequence up to 
round t, and let Ej be the 2nd-moment matrix of the reward features. 

Lemma 18. With probability at least 1 — 5, for all t <T, 


(tv - 4:Ly/tV\n{4:LT/5) - 4Lln{4:LT/S)j II, 


where is the L x L identity matrix. 

Proof. For iT = 1, we have L = 0, so the bound holds. In the remainder, assume K > 2. The proof has two 
components: the spectral decomposition of the mean EEj and the deviation bound on E^. 

Spectral decomposition of EE*: The first step in the proof is to analyze the expected value of the 
2nd-moment matrix. Since yt,At are identically distributed, it suffices to consider just one term. Fixing x 
and y, we only reason about the randomness in picking A. Let S := E^^c/ [yiA)y{A)'^] G be the 

mean matrix for that round. We have: 



Define y := A v[a), Ey := ’= ~ 2/^’ observe that by the definition 

of V in Eq. (22), we have Rx.yVy = V. Continuing the derivation, we obtain: 



To finish the derivation, let u = 1 /s/L be the unit vector in the direction of all ones and P = I — uvA be the 
projection matrix on the subspace orthogonal with u. Then 
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Thus, 

s = + is’) 

By taking the expectation, we obtain the spectral decomposition with eigenvalues A„ and A p associated, 
respectively, with and P: 

+ mf]^ P- ( 25 ) 

-V-^ ^-^ 

Au A p 


We next bound the eigenvalue An. By positivity of y, note that Ey < (max^ y{a))y < Ky^. Therefore, 
Vy = Ey — y^ < {K — l)y^, and thus E[y^] > Vf{K — 1), so 


A 


U 


K-L 
K- 1 


F + m[f] > 


K 

K-1 


V. 


Thus, both eigenvalues are lower bounded by V > V. 

The deviation bound: For deviation bound, we follow the spectral structure of ESj and first reason 
about the properties of Eju, followed by the analysis of PEjP. Throughout the analysis, let Zi := yi(Ai) 
denote the T-dimensional reward feature vector on round i, and consider a fixed t < T. 

Direction u: We begin by the analysis of Stu. Specifically, we will show that ||Etu — (EEt)u ||2 is small. 
We apply Bernstein’s inequality to a single coordinate i, then take a union bound to obtain a bound on H-Hoo, 
and convert to abound on 11-112. For a fixed i and i < t, define 

■= Zitzfu 


and note that {'Etu)e = J2i<t range and variance of Xi are bounded as 

0 < Wi < /l 

E[Xf] = E[4(zfu)2] < Eazfuf] = u^E[z.zf]u = A. 


where the last equality follows by Eq. (25). Thus, by Bernstein’s inequality, with probability at least 1 — 6/2L, 




< y/2tXu HAL/6) + /Lln(4L/(5)/3. 


Taking a union bound over i < L yields that with probability at least 1 — S/2, 

||EtM- (EEt)M ||2 < VL\\EtU - (EEt)u\\^ < yj2LtXu ln{4L/d) + L ln(4L/5)/3. (26) 

Orthogonal to u: In the subspace orthogonal to u, we apply the matrix Bernstein inequality. Let X^, for 
i < f, be the matrix random variable 

X, := Pz,zf P - PE[z,zf]P 

and note that J2i<t Xi = PEtP — E[PEtP]. Since Zi are i.i.d., below we analyze a single Zi and Xi and 
drop the index i. The range can be bounded as 

Amax(-^) < X^^^iPzz'^P) < \\z\\2 < L. 
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To bound the variance, we use Schatten norms, i.e., Lp norms applied to the spectmm of a symmetric matrix. 
The Schatten p-norm is denoted as 11 • 11 0 -,^. Note that the operator norm is | j • | j cr,oo and the trace norm is 11 • | j < 7 , 1 . 
We begin by upper-bounding the variance by the second moment, then use the convexity of the norm, the 
monotonicity of Schatten norms, and the fact that the trace norm of a positive semi-definite matrix equals its 
trace to obtain; 




< 


< E 


E[{Pzz^Pf] 
[Pzz'^Pfl 


< E 


\{Pzz^PY\U,, 


= E 


trUPzz'^Pf 


and continue by the matrix Holder inequality, tr(A^i?) < |j A|jcr_oo||5||o-,i, and Eq. (25) to obtain: 


< (maxIlPz^^PlI 


E 


\PzZ^P\\a,l 


< LtrE[P zz^P] = L{L — l)Ap. 


Reverting to the notation || • || for the operator norm, the matrix Bernstein inequality (Lemma 26) yields that 
with probability at least 1 — 5/2, 


PE(P-E[PEtP] =||^X,-^EXi < ^j2LH\phi{2L/5)+2L\n{2L/5)/?,. (27) 






The final bound: Let x be an arbitrary unit vector. Decompose it along the all-ones direction and the 
orthogonal direction as a; = au -I- fiv, where v P u, and + fP = 1. Let v' = (Ej — EEt)tt. Then 




EEt)a: 


au^{Et — EEt)a; -I- I3v^{Et — EEt)aM -|- jSv^{Et — EEt)/3u 


< |a| • ||u^(Et - EEt)||2 + \aP\ ■ || (E* - EE^^H^ + |u^(Et - EEt)v 
<2|a|.||u'||2 + |/3|-||^^SiP-E[PEtP]|| . 


(28) 


From Eq. (25), we have 

a;^(EEt)a: > a^tXu + 0^t\p. (29) 

To finish the proof, we will use the identity valid for all yl, P, c > 0 

A + B - cpA + B > A + B - cY~A - cVb + c^/4 - c^/4 
= A- cVA + (Tp - c/2)^ - P/4 
> A- cVA - P/4. (30) 


Combining Eq. (28) and Eq. (29), and plugging in bounds from Eq. (26) and Eq. (27), we have 

a;^Eta; > aYXu + pHXp - 2|a| • ||u'||2 - |/3| • ||PEtP - E[PEtP]|| 

> ahXu + phXp - 2\aW2LtXu \p4L/5) - ln(4L/b) 

- \l3W2LHXp \^{2L/5)- ln(2L/,5) 

> aHXu + PHV - {2yj2L\n{4L/5)^ \/aHXu 

- |/3| ^j4LHV\p4L/5) - 2L \tl{4L/5), 
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where we used V < Xp < 2V, and |a| < 1, |/3| < 1. We now apply Eq. (30) with A + B = a^tX^ and 
A = a^tV to obtain 

x^T^tx > aHV + - {2^2L\ii{AL/5)^ VaHV - 2L\n{4L/S) 

- 2L\/3\^ytV\n{4L/S) - 2Lln{4L/6) 

>tV- 2LV2 (|a| + |;3|) y/tVln{4L/S) - 4Lln(4L/5) 

>tV- 4:Ly/iV\^iLj^ - AL \tl{AL/5), 

where we used |a| + |/3| < \/2a^ + 2/3^ = -^2. The lemma follows by the union bound over t <T. □ 


F.4 Analysis of the Exploration Regret 

The analysis here is made complicated by the fact that the stopping time of the exploration phase is a random 
variable. If we let t denote the last round of the exploration phase, this quantity is a random variable that 
depends on the history of interaction up to and including round t. Our proof here will use a non-random 
bound t* that satisfies P(f < f*) > 1 — i5. We will compute t* based on our analysis of the 2nd-moment 
matrix Ej. 

A trivial bound on the exploration regret is 




rt{At) 


<t*\\w*hVL, 


(31) 


which follows from the Cauchy-Schwarz inequality and the fact that the reward features are in [0,1], 

In addition, we also bound the exploration regret by the following more precise bound: 

Lemma 19 (Exploration Regret Lemma). Let t* be a non-random upper bound on the random variable t 
satisfying P(i < t*) > 1 — 5. Then with probability at least 1 — 25, the exploration regret is 




rtiAt 


< f*||w*||2min. 


Ww^hV'^Lt* ln(l/5). 


Proof. Let {xt,yt,At,^t}t=i be a sequence of random variables where {x,y,^) ~ D and At is drawn 
uniformly at random. We are interested in bounding the probability of the event 


£ := 



This term is exactly the exploration regret, so we want to make sure the probability of this event is large. We 
first apply the upper bound 


-ytiAt)j w* <^[yt{A*) - yt{At)^ w*, 
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where = arg maxyi yt {A)’^w* is the best possible ranking. This upper bound ensures that every term in 
the sum is non-negative. We next remove the dependence on the random stopping time t and replace it with a 
deterministic number of terms t*: 


P(£) > P - yt{At)Yw* < e 

( i j, 

- yMt)') w^<e n i<t* 

> P - yt(^)) V < e n f < f*j 

> 1 ^ - P (f > t*) 

ft* ^ \ 


> i-,5-: 


> e . 


The first line follows from the definition of A^ which only increases the sum, so decreases the probability 
of the event. The second inequality is immediate, while the third inequality holds because all terms of the 
sequence are non-negative. The fourth inequality is the union bound and the last is by assumption on the 
event {£ < t*}. 

Now we can apply a standard concentration analysis. The mean of the random variables is 


E 


x^y^A 


{y{A*) - yiA)fw* < ||u ;*||2 [j/(A*) - j/(A)] 


= \\w *\\2 r^^x,y [y{A*)-yY 


KL 


<\\w*h r^^x,y {y{AY)-yy 


KL 


< ||w*||2V^J—(y(a)-J/)' 


o-G-A 


= llt« II 2 




The first inequality is Cauchy-Schwarz while the second is Jensen’s inequality and the third comes from 
adding non-negative terms. The range of the random variable is bounded as 


sup 

x,y,A 


(yiA*) - yiA))\* - E^^yJ{yiA*) - yiA))\* 


< Ikibx/T, 


because 0 < (y{A*) — y{A))^w* < ||■u;*|| 2 ^/T. Thus by Hoeffding’s inequality, with probability at least 
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l-<5, 


Y^iviA*) - yiA)fw* < - 2 /(A)) V] + ||zz ;*||2 V2Li* ln(l/<5) 

t^l t^l 

< t*\\w*\\ 2 Vl^+\\w*\\ 2 V^Lt*\n{l/S). 

Combining this bound with the bound of Eq. (31) proves the lemma. 


□ 


F.5 Analysis of the Exploitation Regret 

In this section we show that after the exploration rounds, we can find a policy that has low expected regret. 
The technical bulk of this section involves a series of deviation bounds showing that we have good estimates 
of the expected reward for each policy. 

In addition to y from the previous sections, we will also need the sample quantity yt := 'Yha&A 
which will allow us to relate the exploitation regret to the variance term V. Since we are using uniform 
exploration, the importance-weighted feature vectors are as follows; 


_ l(a € At)yt{a) 
U{a (z Af) 


K 

~L 


l(a e At)yt{a). 


Given any estimate w of the true weight vector w*, the empirical reward estimate for a policy tt is 


??n(7r, W) 




A natural way to show that the policy with a low empirical reward has also a low expected regret is to show 
that for all policies tt, the empirical reward estimate ?7„(7r, w) is close to the true reward, r]{TT), defined as, 

?7(7r) := [y{TT{x)fw*] . 

Rather than bounding the deviation of ? 7 „ directly, we instead control a shifted version of p„, namely. 


tpn{Tr,w) 


1 j T 

- Y [yt(7r(a:t))'^w - ytl^w] , 
n ^' 

t=i 


where 1 is the L-dimensional all-ones vector. Note that yt is based on the rewards of all actions, even those 
that were not chosen at round t. This is not an issue, since yt is only used in the analysis. 

Lemma 20. Fix 5 € (0,1) and assume that Hw — 1/7*112 < d for some 0 > 0. For any 5 S (0,1), with 
probability at least 1 — 5, we have that for all tt S IT, 


|V'n(7r, w) - r]{n, w*) + [yl'^w*] \ 

< 2(0 + ||zu*||2)v^ + \/f ) + ^inin{ViTy, 2 ^ 1 }. 
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Proof. We add and subtract several terms to obtain a decomposition. We introduce the shorthands := 

y{Tr{x)), := y{Tr{x)), and := yt{Tr{xt)). 

■i/'n(7r, w) - ?7(7r, w*) + [yl'^w*] 

1 " 

= - '^{yt,^ - - ^x,y[y^ - 

^ t=i 


n 

^(^iyt,n - yti)'^w - E^^y[y^ - yl]'^w^ + E^^y[y^ - yl]'^{w - w*). 


t=i 


Term 1 


Term 2 


There are two terms to bound here. We bound the hrst term by Bernstein’s inequality, using that fact that ijt is 
coordinate-wise unbiased for y. The second term will be bounded via a deterministic analysis, which will 
yield an upper bound related to the reward-feature variance V. 

Term 1: Note that each term of the sum has expectation zero, since yt is an unbiased estimate. Moreover, 
the range of each individual term in the sum can be bounded as 


(yt.rr - ytl - Ex.ybrr - yl])^W <l|w>l |2 ^i.Tr “ 2/i 1 “ - 1/1] 

‘IK 

<{e + \\w*h)^. 


The second line is derived by bounding the two factors separately. The hrst factor is bounded by the triangle 
inequality: ||r()||2< ||iu*||2-l-||ih — ii'*||2< ||■u;*||2-l-^*. The second factor is a norm of an L-dimensional 
vector. The vector yt has coordinates in [0, K/L], whereas the coordinates of ytl, y 7 r, and yl are all in 
[0,1], so the hnal vector has coordinates in [—2, K/L + 1], and its Euclidean norm is thus at most •\/L(2iT/L) 
since K > L. 

The variance can be bounded by the second moment, which is 


E, 


x,y.,A 


((y^ - ylfw) 


< IIWII 9 E; 


2^x,y,A 


< llu-II^E. 


2^x,y,A 
L 


- yf 

L 

'^{y{TT{x)if +y- 


. 1=1 


1=1 

L 


K 




t=i 


= ll^lli 51 ( 7-Ex,y[y(7r(a:)£)] -f 1 


<2{e+\\w*hfK, 


where the last inequality uses K > L. Bernstein’s inequality implies that with probability at least 1 — (5, for 
all TT € If, 


n 

5^((yt,77 - ytl)^W - Ea;,y[y^ - 


<(0 + lk*ll2) 


AK\ii{2N/5) 


AK\n{2N/5) 

Su'/L 
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Term 2: For the second term, we use the Cauchy-Schwarz inequality, 

^x,y[y^ - - '*"*) < ||Ea;,y[y,7 - ylJiyiti - w *\\2 

The difference in the weight vectors will be controlled by our analysis of the least squares problem. We need 
to bound the other quantity here and we will use two different bounds. First, 

||Ea:,y[y7r “ yl]||2 < - yl||2 < E^^^y ||||2 + y||l||2 < ‘2'/L. 

Second, 


\\^x,y[y^ - 2 / 1 ] II 2 = 


\ 


^x,y XI {yi'^i^)t) -y) ^ X 


r=i 


fiG*A 


= VKjE^^y- X(2/(a) - y? = 




Combining everything: Putting everything together, we obtain the bound 

+ 6 iJim{VKV, 2 Vl}. 


(0+lk*l|2) 


4iFln(2iV/,5) AKlii{2N/S) 


iri'/L 


Collecting terms together proves the main result. 


□ 


Assume that we explore for i rounds and then call AMO with weight vector w and importance-weighted 
rewards yi,.. .y^ to produce a policy -if that maximizes r]f.{TT, w). In the remaining exploitation rounds we act 
according to tt. With an application of Lemma 20, we can then bound the regret in the exploitation phase. 
Note that the algorithm ensures that t is at least equal to the deterministic quantity n*, so we can remove the 
dependence on the random variable t: 

Lemma 21 (Exploitation Regret Lemma). Assume that we explore for t rounds, where t > n*, and we find w 
satisfying |jw — U '*||2 < Then for any 6 € (0,1), with probability at least 1 — 2i5, the exploitation regret is 
at most 


i=£+l 


rt{TT{xt)) <4T{9 + \ 


2)VK 


|^^/ ln(2W/J) 


[k\w{2N/5) 

V L 


+ 2Termii{s/KV, 2 s/L} + ||w*||2\/2Lrin(l/(5). 


Proof Using Lemma 20 and the optimality of w for the importance-weighted rewards, with probability at 
least 1 — 5, the expected per-round regret of if is at most 

ri{'K*, w*) — r]{TT, w*) 

= [??(7r*,w*) -'0((7r*,-u))] -b [fif{Tr,w) + [r]fi7r*,w) 

< 4(0 + ||u;*|!2)v^ ywm ^ ^ 20min{Vi^, 2Vl}. 

To bound the actual exploitation regret, we use Hoeffding’s inequality together with the fact that the absolute 
value of the per-round regret is at most ||r(;*|| 2 '\/L, and finally apply bounds 1/Vt < \j and 1/f < 1/n* 
to prove the lemma. □ 
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F.6 Proving the Final Bound 

The final bound will follow from regret bounds of Lemmas 19 and 21 . These bounds depend on parameters 
t*, n* and 6. The parameter n* is specified directly by the algorithm and is assured to be a lower bound 
on the stopping time. The parameter t* needs to be selected to upper-bound the stopping time t, and 9 to 
upper-bound Ijw) — w*|| 2 - 

The stopping time bound t* and error bound 6: Our algorithm uses the constants 

A* := max|6L2ln(4Lr/5), [TV/Bf^^ (Lln(2/b))^/^} , 
n* := T2/3(iTln(Ar/(5))i/3max{L-i/3^ 

and we will show we can set 

t* := max{6A*/l/, n*} , 9 := ■>/ cL ln(2/(5)/A*, 

where c is the constant from Lemma 17. 

Recall that we assume T > {K \n{N/5)/L) max{l, {By/L)~^'\, which ensures that T > n*, and that the 
algorithm stops exploration with the first round t such that t > rii, and Ainin(S() > A*. Thus, by Lemma 17, 
9 is indeed an upper bound on ||r() — w*|| 2 - Furthermore, since t* > n*, it suffices to argue that E** ^ A*/^ 
with probability at least 1 — 5. We will show this through Lemma 18. 

Specifically, Lemma 18 ensures that after t* rounds the 2nd-moment matrix satisfies, with probability at 
least 1 — b. 


Et* h (t*V - 4:L^yt*V\n{4LT/6) - 4Lln(4Lr/5)) II- 
It suffices to verify that the expression in the parentheses is greater than A*: 

(t*V - 4LVt*Lln(4LT/(5) - 4L ln(4LT/5)) > A* 

(y¥v - 2Ly^\ii{4LT/d)j ^ - 4L^ ln(4LT/b) - 4L ln(4Lr/b) > A* 

(vW - 2LVln(4LT/(5))^ > A* -f ln(4LT/(5) 

V¥V - 2LyJ\ii{4LTl5) > yjK + 8L2 ln(4Lr/b) 

( VA* -f 8L2 ln(4LT/(5) -f 2L Vln(4Lr/b)) ^ 

Our setting is an upper bound on this quantity, using the inequality (a -f b)^ < 2a^ + 2b^ and the fact that 

A* > 6L‘^\n{4LT/5). 

Regret decomposition: We next use Lemmas 19 and 21 with the specific values of t*, n* and 9. The 
leading term in our final regret bound will be on the order In the smaller-order terms, we ignore 

polynomial dependence on parameters other than T (such as K and L), which we make explicit via Ot 
notation, e.g., LT) = OriVT). 

The exploration regret is bounded by Lemma 19, using the bound t* < 6A*/L -f n*, and the fact that the 


40 











exploration vacuously stops at round T, so t* can be replaced by minji*, T}: 

Exploration Regret < min{i*, T}||r(;*|j 2 min{-\/i^E,-s/Z} + 11^*112^2^7^ln(l/(5) 

< min I r| B min{-\/ KV, Vl} + ni,BVL +Or(-\/r). 


Term 1 


Term 2 


Meanwhile, for the exploitation regret, using the fact that n* = Lemma 21 yields 

Exploitation Regret < AT{9 +\\w*\\ 2 )Vk j 

+ 2r6»min{v^FE, 2v^} + ||u>*||2 V2Lrin(l/,5) 

= 4r(0 + Ik*||2) W + 2T0min{y7^, 2^1} + Ot(Vt) 

y n* 


< 4r 


'cLln(2/5) 

a: 


B 


I KIti{2N/S) 


Term 3 


2ry^^^^^min{y7^, 2^1} +OTiVf). 


Term 4 


We now use our settings of n* and A* to bound all the terms. Working with A* is a bit delicate, because it 
relies on the estimate V rather than V. However, by Lemma 16 and Eq. (23), we know that 

E < y < 4E + T, 


where r := 5 ln(2/(5)/(2n*). 

Term 1: We proceed by case analysis. Eirst assume that V < t. Then 


Term 1 < TBVk^ < TB\ ^ 

V 2n* 


so we can use the bound on Term 3 to control this case. 

Next assume that V > t, which implies V < 5V, and distinguish two sub-cases. Eirst, assume that A* is 
the second term in its dehnition, i.e.. A* = {TV {L ln(2/(5))^^^. Then: 


Term 1 < 

< 


min I vZeE, '/l'^ 

6B{TV/Bf/^{L ln(2/(5))i/3 min{/iTE, ^/L} 
V 


< 18Bi/3T2/3E-i/3(Lln(2/5))^/^ min{/^, vT}, 
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where the last step uses We now show that the term involving V and the min{-} 

is always bounded as follows: 

Claim 22. min{\/KV, VL} < 

Proof. If KV < L, then V < LjK, and the expression equals V~^/^\/KV = . On 

the other hand, if L < KV, then V > L/K, and the expression is equal to . □ 

Thus, in this case. Term 1 is O (T^/^L^/^(_BiTlog(2/(5))^/^). 

Finally, assume that A* is the first term in its definition, i.e., 

A* = ln(4LT/(5) > {TV (L ln(2/(5))^/^ , 

which implies 

y < y < in(4LT/(5)3/2(B/T)(ln(2/5))-i/2^ (32) 

Thus, we have the bound 

Term 1 < = O \og{LT/5)^ = Ot{,Vt). 

In summary, we have the bound. 

Term 1 < Term 3 + 0 {t^/^{BK\ og{2 /+ Ot{Vt). (33) 

Term 2: Plugging in the definition of n* yields 

Term 2 = n*Bv^ < ^i/3^-i/6|_ ( 34 ^ 

Term 3: Note that 

1/^ = r-i/3(inn(iV/,5))-^/® min{Li/®, {BLy/^}, 

A* > 6+2 ln(4LT/(5) > L‘^\n{2/6), 


so 


Term 3 = 4+ 


lcL\n(2/S) 


A* 


B 


lKln{2N/S) 


< O ^+2/3 {K\n{N/6)y^^min{L^/^,{BLy/^} 


Now if B - Vl , then the min above is achieved by the T^/e term, so the bound is 


O {T^/^BL^/^{K\og{N/5)f/^^ . 


IfB - Vl , then the min is achieved by the {BLf/^ term, so the bound is 


O {T^/^B^/^L-^/^{K\og{N/5)y/^^ . 
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Thus, 


Term 3 = 


3 = 0 (T^/^{K\og{N/6)y/^ 


(35) 


Term 4: We distinguish two cases. If A* = 6L^ ln(4LT/(5) then Eq. (32) holds and thus 


V = 0 ln(4LT/(5)3/2(B/r)^ . 


We then have 


Term 4 = 2T^ 


< O = O {^BTKL^!^ ln(Lr/5)) = Ot{Vt). 

Otherwise, A* = {TVj (Lln(2/i5))^^^, and since lA > E, we obtain 


Term 4 = min{v/iTE, 2Vl} 

= O (T2/3Bi/32,i/3y-i/3(iog(2/j))i/3 uim{y/KV, Vl}^ 

= O (T2/3(BiT)i/32,i/2(iog(2/^))i/3^ ^ (36) 

where the last step is Claim 22. This is the leading-order term since the other cases are Ot{VT). 

Putting everything together: Combining Eqs. (33), (34), (35), and (36), we obtain the bound on the 
sum of the exploration and exploitation regret; 

Regret = O (T^/^{Klog{N/d))^/^ , BL^/^}^ + Ot{Vt). 


G Deviation Bounds 


Here, we collect several deviation bounds that we use in our proofs. All of these results are well known and 
we point to references rather than provide proofs. The first inequality, which is a Bernstein-type deviation 
bound for martingales, is Ereedman’s inequality, taken from Beygelzimer et. al [4] 

Lemma 23 (Freedman’s Inequality). Let Xi, X 2 ,..., Xt be a sequence of real-valued random variables. 
Assume for all t £ {1, 2,..., T} that Xt < R and E[Ar(|Aii,..., Xt-i\ = 0. Define S = X]t=i 
V = X)t=i ^ ^t-i\ For any 5 £ (0,1) and A £ [0,1/i?], with probability at least 1 — 6: 


S< {e-2)XV + 


ln(l/^) 

A 
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We also use Azuma’s inequality, a Hoeffding-type inequality for martingales. 

Lemma 24 (Azuma’s Inequality). Let Xi, X 2 ,... be a sequence of real-valued random variables. 
Assume for all t G {1, 2,..., T} that Xt < R and E[Xt |Xi,..., Xt-i] = 0. Define S = For any 

5 € (0,1), with probability at least 1 — 5: 

S < Rsj2T\w{l/5) 

We also make use of a vector-valued version of Hoeffding’s inequality, known as the Hanson-Wright 
inequality, due to Rudelson and Vershynin [23], 

Lemma 25 (Hanson-Wright Inequality [23]). Let X = (Xi,..., X„) be a random vector with independent 
components satisfying EX^ = 0 and \Xi\ < k almost surely. There exists a universal constant cq > 0 such 
that, for any A € and any 6 € (0,1), with probability at least 1 — <5, 

IX^AX - EX^AXI < K^sJcc,\\A\\l\og{2/5) + cok^\\A\\ log(2/5), 

where || • Up- is the Frobenius norm and H-H is the spectral norm. 

Finally, we use a well known matrix-valued version of Bernstein’s inequality, taken from Tropp [26]. 

Lemma 26 (Matrix Bernstein). Consider a finite sequence {X^} of independent, random, self-adjoint 
matrices with dimension d. Assume that for each random matrix we have EX^ = 0 and Amax(-^fe) ^ R 
almost surely. Then for any 6 S (0,1), with probability at least 1 — 5: 

Amax(^^fc) < s/2ann{d/5) + ‘^R\og{d/5) with = ||;^E(X2)||, 

k k 

where H-H is the spectral norm. 
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